Nienke van Atteveldt|Micah M. Murray|Gregor Thut|Charles E. Schroeder
Multisensory Integration: Flexible Use of General Operations
Maastricht University|Royal Netherlands Academy of Arts and Sciences|Netherlands Institute for Neuroscience|Amsterdam Neuroscience|Centre d'Imagerie BioMedicale|University of Lausanne|University of Glasgow|Nathan Kline Institute for Psychiatric Research|New York Psychoanalytic Society and Institute|Columbia University
research anatomical substrates principles integrating inputs separate sensory surfaces yielded divergent findings suggests multisensory integration flexible context dependent underlines need dynamically adaptive neuronal integration mechanisms propose flexible multisensory integration explained combination canonical populationlevel integrative operations oscillatory phase resetting divisive normalization canonical operations subsume multisensory integration fundamental set principles brain integrates sorts information used proactively adaptively illustrate proposition unifying recent findings different research themes timing behavioral goal experiencerelated differences integration research anatomical substrates principles integrating inputs separate sensory surfaces yielded divergent findings suggests multisensory integration flexible context dependent underlines need dynamically adaptive neuronal integration mechanisms propose flexible multisensory integration explained combination canonical populationlevel integrative operations oscillatory phase resetting divisive normalization canonical operations subsume multisensory integration fundamental set principles brain integrates sorts information used proactively adaptively illustrate proposition unifying recent findings different research themes timing behavioral goal experiencerelated differences integration perception generally multisensory process situations involve sight sound perhaps touch taste smell sensory input acquired least modulated motor sampling strategies routines schroeder et al 2010schroeder ce wilson da radman scharfman h lakatos p dynamics active sensing perceptual selectioncurr opin neurobiol 2010 20 172176crossref pubmed scopus 152 google scholar perception also sensorimotor process brain must constantly combine kinds information moreover track anticipate changes one cues cues different sensory modalities initially enter nervous system different ways historically within modality crossmodal integration studied separately fields lately converged leading suggestion neocortical operations essentially multisensory ghazanfar schroeder 2006ghazanfar aa schroeder ce neocortex essentially multisensorytrends cogn sci 2006 10 278285abstract full text full text pdf pubmed scopus 660 google scholar following notion integration essential part sensory processing one might presume matter whether different inputs onto neuron say primary auditory cortex come modality consider multisensory processing focus cortical auditoryvisual processing representative brains integrative operations general extension consider multisensory integration model rather empirical probe provides unique window integrative brain adaptive nature multisensory paradigms provide inputs onto integrative machinery brain segregated different receptor surfaces initial input pathways facilitates clean identification initial point convergence something difficult within main sensory modalities framework brain bayesian estimator environment ernst banks 2002ernst mo banks ms humans integrate visual haptic information statistically optimal fashionnature 2002 415 429433crossref pubmed scopus 1841 google scholar fetsch et al 2013fetsch cr deangelis gc angelaki de bridging gap theories sensory cue integration physiology multisensory neuronsnat rev neurosci 2013 14 429442crossref pubmed scopus 79 google scholar knill pouget 2004knill dc pouget bayesian brain role uncertainty neural coding computationtrends neurosci 2004 27 712719abstract full text full text pdf pubmed scopus 679 google scholar uncertainty sensory estimates minimized combining multiple independent measurements exactly multisensory integration might play important role multisensory cues often provide complementary estimates event withinmodality cues tend less equally reliable example dusk audition vision provide signals complementary strength whereas visual shape texture cues degraded moreover cues different modalities singular objects events often predict typically crosscorrelate one sense lags parise et al 2012parise cv spence c ernst mo correlation implies causation multisensory integrationcurr biol 2012 22 4649abstract full text full text pdf pubmed scopus 0 google scholar whereas withinmodality cues similar timing words interpreting multisensory withinsensory cue pairings considered different means reducing uncertainty event within wider class integrative processes search specific underpinnings multisensory integration findings diverse example regard role binding principles spatial temporal correspondence inverse effectiveness stein meredith 1993stein meredith merging senses mit press cambridge1993google scholar principle spatial correspondence simplified version states inputs likely integrated overlap space might dependent task requirements previously thought cappe et al 2012cappe c thelen romei v thut g murray mm looming signals reveal synergistic principles multisensory integrationj neurosci 2012 32 11711182crossref pubmed scopus 0 google scholar girard et al 2011girard collignon lepore f multisensory gain within across hemispaces simple choice reaction time paradigmsexp brain res 2011 214 18crossref pubmed scopus 14 google scholar sperdin et al 2010sperdin hf cappe c murray mm auditorysomatosensory multisensory interactions humans dissociating detection spatial discriminationneuropsychologia 2010 48 36963705crossref pubmed scopus 0 google scholar furthermore importance temporal proximity seems differ across stimulus types tasks stevenson wallace 2013stevenson ra wallace mt multisensory temporal integration task stimulus dependenciesexp brain res 2013 227 249261crossref pubmed scopus 0 google scholar van atteveldt et al 2007avan atteveldt nm formisano e blomert l goebel r effect temporal asynchrony multisensory integration letters speech soundscereb cortex 2007 17 962974crossref pubmed scopus 0 google scholar principle inverse effectiveness context dependent predicts stimulus intensity primary determinant integration effects stein meredith 1993stein meredith merging senses mit press cambridge1993google scholar moreover use multiple anatomical substrates integration also likely context dependent example different potential sources multisensory influences lowlevel sensory cortices put forward including direct crossing projections sensory cortices different modalities eg falchier et al 2002falchier clavagnier barone p kennedy h anatomical evidence multimodal integration primate striate cortexj neurosci 2002 22 57495759crossref pubmed google scholar rockland ojima 2003rockland ks ojima h multisensory convergence calcarine visual areas macaque monkeyint j psychophysiol 2003 50 1926crossref pubmed scopus 343 google scholar direct ascending inputs socalled nonspecific thalamic regions hackett et al 2007hackett ta de la mothe la ulbert karmos g smiley j schroeder ce multisensory convergence auditory cortex ii thalamocortical connections caudal superior temporal planej comp neurol 2007 502 924952crossref pubmed scopus 0 google scholar schroeder et al 2003schroeder ce smiley jf fu kg mcginnis oconnell mn hackett ta anatomical mechanisms functional implications multisensory convergence early cortical processingint j psychophysiol 2003 50 517crossref pubmed scopus 0 google scholar well indirect feedback inputs higherorder multisensory cortical regions smiley et al 2007smiley jf hackett ta ulbert karmas g lakatos p javitt dc schroeder ce multisensory convergence auditory cortex cortical connections caudal superior temporal plane macaque monkeysj comp neurol 2007 502 894923crossref pubmed scopus 0 google scholar several accounts multiple architectures may functionally complementary driver noesselt 2008driver j noesselt multisensory interplay reveals crossmodal influences sensoryspecific brain regions neural responses judgmentsneuron 2008 57 1123abstract full text full text pdf pubmed scopus 460 google scholar ghazanfar schroeder 2006ghazanfar aa schroeder ce neocortex essentially multisensorytrends cogn sci 2006 10 278285abstract full text full text pdf pubmed scopus 660 google scholar van atteveldt et al 2013van atteveldt nm peterson bs schroeder ce contextual control audiovisual integration lowlevel sensory corticeshum brain mapp 2013 published online august 24 2013httpsdoiorg101002hbm22336crossref pubmed scopus 6 google scholar werner noppeney 2010werner noppeney u distinct functional contributions primary sensory association areas audiovisual integration object categorizationj neurosci 2010 30 26622675crossref pubmed scopus 93 google scholar apparent flexibility integrative principles neuronal architectures indicates multisensory integration fixed uniform strongly adaptive contextual factors idea contextdependent integration new especially within aforementioned view brain statistically optimal cue integrator example modalityappropriateness framework context determines senses provides appropriate reliable information welch warren 1980welch rb warren dh immediate perceptual response intersensory discrepancypsychol bull 1980 88 638667crossref pubmed scopus 568 google scholar general visual cues reliable spatial judgments whereas sounds provide reliable temporal cues either modality dominate perception different contexts short multiple factors input properties behavioral goal dynamically flexibly interact provide momentary context composite context seems adaptively recruit different neuronal operations pathways integration based recent neurophysiological findings detailed take perspective multisensory integration subsumed canonical integrative operations unified view diversity research findings view reflects flexible use canonical operations substantiate perspective section 2 first consider two exemplar canonical integrative operations detail divisive normalization dn oscillatory phase resetting pr illustrate combination canonical operations may enable observed highly adaptive contextdependent nature brains integrative processing rather providing exhaustive review reported context effects focus three main themes within multisensory research timing behavioral relevance effects experience accordingly section 3 discuss temporal predictability influences brains operation mode illustrate potential complementary role dn pr different modes section 4 focus different behavioral goals task sets guide flexible use canonical integrative operations section 5 address integrative processing shaped shortterm longerterm changes sensory context related experience training noted another main themeie attentioninfluences processing within themes therefore frame attention working concert suggested canonical integrative operations discuss role whenever relevant explaining operations subsume adaptive integration within different main themes finally discuss whether certain integrative processes less context dependent others final notes mean context worthwhile review use context immediate situation brain operates context shaped external circumstances properties sensory events internal factors behavioral goal motor plan past experiences fact internal external factors often interactively define context example structure relevant sensory events switches brain specific internal operation mode rhythmic mode schroeder lakatos 2009schroeder ce lakatos p lowfrequency neuronal oscillations instruments sensory selectiontrends neurosci 2009 32 918abstract full text full text pdf pubmed scopus 553 google scholar longerlasting determinants context refer individual experiences shape integrative operations developmental context viewpoint therefore basic though mutually exclusive use context voluntary psychological function ability contextualize information purpose assigning value events maren et al 2013maren phan kl liberzon contextual brain implications fear conditioning extinction psychopathologynat rev neurosci 2013 14 417428crossref pubmed scopus 362 google scholar neuronal bases sensory integration formed 1 convergence synaptic inputs multiple sources onto individual neurons 2 operation neurons perform produce integrated output signals 3 interactions neurons within across populations networklevel interactions initial integration process converging inputs originate different sensory modalities moreover sensory inputs also need combined ongoing motor actions well topdown signals relate current inputs knowledge memories predictions review advance general neurophysiological framework designed account wide variety integrative processes brain constantly performs prior studies suggested within across modality integration different behavioral forster et al 2002forster b cavinapratesi c aglioti sm berlucchi g redundant target effect intersensory facilitation visualtactile interactions simple reaction timeexp brain res 2002 143 480487crossref pubmed scopus 0 google scholar gingras et al 2009gingras g rowland ba stein differing impact multisensory unisensory integration behaviorj neurosci 2009 29 48974902crossref pubmed scopus 0 google scholar girard et al 2013girard pelland lepore f collignon impact spatial congruence redundant targets withinmodal crossmodal integrationexp brain res 2013 224 275285crossref pubmed scopus 10 google scholar neurophysiological alvarado et al 2007alvarado jc vaughan jw stanford tr stein multisensory versus unisensory integration contrasting modes superior colliculusj neurophysiol 2007 97 31933205crossref pubmed scopus 54 google scholar underpinnings question whether due unique intrinsic properties multisensory neurons integrative computations differences multiple convergent cues typically interact within versus across senses ie way provide different estimates event locally wired latter case multisensory integration special essence information provided different modalities may lead stronger neuronal interactions information tends complementary ability reduce uncertainty events singleneuron level superior colliculus sc found indeed input determines response neuron type unisensory multisensory alvarado et al 2007alvarado jc vaughan jw stanford tr stein multisensory versus unisensory integration contrasting modes superior colliculusj neurophysiol 2007 97 31933205crossref pubmed scopus 54 google scholar compared visualvisual integration visualauditory integration multisensory unisensory neurons cat sc visualvisual integration found subadditive integrative response multisensory unisensory neurons audiovisual integration occurs multisensory neurons response different namely additive superadditive computational models group explain different responses multisensory sc neurons different clustering synaptic inputs alvarado et al 2008alvarado jc rowland ba stanford tr stein neural network model multisensory integration also accounts unisensory integration superior colliculusbrain res 2008 1242 1323crossref pubmed scopus 0 google scholar rowland et al 2007rowland ba stanford tr stein model neural mechanisms underlying multisensory integration superior colliculusperception 2007 36 14311443crossref pubmed scopus 0 google scholar inputs cluster together dendritic unit neuron case multisensory inputs produce stronger synergistic interaction compared inputs cluster together noted singlecell interactions may determinative structures like sc neocortex instead substantiate ensemble processes provide degrees freedom flexibility differing contexts recently divisive normalization model developed visual processing carandini et al 1997carandini heeger dj movshon ja linearity normalization simple cells macaque primary visual cortexj neurosci 1997 17 86218644crossref pubmed google scholar reynolds heeger 2009reynolds jh heeger dj normalization model attentionneuron 2009 61 168185abstract full text full text pdf pubmed scopus 521 google scholar described canonical operation carandini heeger 2012carandini heeger dj normalization canonical neural computationnat rev neurosci 2012 13 5162crossref scopus 173 google scholar shown explain important features multisensory integration inverse effectiveness spatial principle fetsch et al 2013fetsch cr deangelis gc angelaki de bridging gap theories sensory cue integration physiology multisensory neuronsnat rev neurosci 2013 14 429442crossref pubmed scopus 79 google scholar ohshiro et al 2011ohshiro angelaki de deangelis gc normalization model multisensory integrationnat neurosci 2011 14 775782crossref pubmed scopus 108 google scholar important feature model integrative outputs normalized surrounding neurons figure 1 thus transcends level single neuron responses model explains integration effects subcortical sc cortical mst measurements interestingly also accounts adaptive changes weighting different inputs function cue reliability morgan et al 2008morgan ml deangelis gc angelaki de multisensory integration macaque visual cortex depends cue reliabilityneuron 2008 59 662673abstract full text full text pdf pubmed scopus 97 google scholar provides neural basis similar effects performance level ernst banks 2002ernst mo banks ms humans integrate visual haptic information statistically optimal fashionnature 2002 415 429433crossref pubmed scopus 1841 google scholar sum networklevel operation divisive normalization able explain cue integration regardless origin cues flexible process depending cue reliability open question framework predictive cue integration accomplished ie cues influence processing future events neural mechanism especially suitable explain predictive interactions phase resetting pr ongoing oscillatory activity kayser et al 2008kayser c petkov ci logothetis nk visual modulation neurons auditory cortexcereb cortex 2008 18 15601574crossref pubmed scopus 231 google scholar lakatos et al 2005lakatos p shah knuth kh ulbert karmos g schroeder ce oscillatory hierarchy controlling neuronal excitability stimulus processing auditory cortexj neurophysiol 2005 94 19041911crossref pubmed scopus 538 google scholar lakatos et al 2007lakatos p chen cm oconnell mn mills schroeder ce neuronal oscillations multisensory interaction primary auditory cortexneuron 2007 53 279292abstract full text full text pdf pubmed scopus 496 google scholar taking primary auditory cortex example shown response amplitudes sounds depend phase ambient oscillations ideal worst phases terms neuronal excitability lakatos et al 2005lakatos p shah knuth kh ulbert karmos g schroeder ce oscillatory hierarchy controlling neuronal excitability stimulus processing auditory cortexj neurophysiol 2005 94 19041911crossref pubmed scopus 538 google scholar predictive influence exerted one event resets phase ongoing excitability fluctuations thereby influences processing upcoming events different modality figures 1 2 phasereset mechanism specific multisensory interactions rather represents general mechanism different sensory motor attentional cues modulate ongoing processing lakatos et al 2013lakatos p musacchia g oconnel mn falchier ay javitt dc schroeder ce spectrotemporal filter mechanism auditory selective attentionneuron 2013 77 750761abstract full text full text pdf pubmed scopus 117 google scholar makeig et al 2004makeig debener onton j delorme mining eventrelated brain dynamicstrends cogn sci 2004 8 204210abstract full text full text pdf pubmed scopus 675 google scholar rajkai et al 2008rajkai c lakatos p chen cm pincze z karmos g schroeder ce transient cortical excitation onset visual fixationcereb cortex 2008 18 200209crossref pubmed scopus 104 google scholar shah et al 2004shah bressler sl knuth kh ding mehta ad ulbert schroeder ce neural dynamics fundamental mechanisms eventrelated brain potentialscereb cortex 2004 14 476483crossref pubmed scopus 0 google scholar memory formation rizzuto et al 2003rizzuto ds madsen jr bromfield eb schulzebonhage seelig aschenbrennerscheibe r kahana mj reset human neocortical oscillations working memory taskproc natl acad sci usa 2003 100 79317936crossref pubmed scopus 166 google scholar therefore propose phase resetting second canonical operation enabling flexible integration multiple sensory motor topdown cuesfigure 2evidence macaque human brain crossmodal phase reset mechanism predictive integrationshow full captiona effect somatosensoryauditory stimulus onset asynchrony soa bimodal response left color map shows eventrelated current source density csd response site large current sink supragranular layers macaque area ai different somatosensoryauditory soas csd index net synaptic responses transmembrane currents lead action potentials indexed concomitant multiunit activity mua signal local neuronal ensemble increasing soas mapped axis top bottom 0 top corresponding simultaneous auditorysomatosensory stimulation au bottom represents auditory alone condition red dotted lines denote 2060 ms time interval averaged csd mua single trials quantification right represent mean csd mua amplitude values x axis 2060 ms auditory poststimulus time interval error bars show se different somatosensoryauditory soas axis stars denote number experiments total six given soa bimodal response amplitude significantly different auditory peaks functions occur soas 27 45 114 976 ms correspond periods oscillations gamma 3050 hz beta 1425 hz theta 57 hz delta 14 hz ranges phase reset thus aligned trials initial somatosensory input csd concomitant mua increases signify increases local neuronal excitation findings illustrate phase reset ongoing oscillatory activity a1 predictively prepares local neurons respond preferentially auditory inputs particular timing relationships somatosensory resetting input reprinted lakatos et al 2007lakatos p chen cm oconnell mn mills schroeder ce neuronal oscillations multisensory interaction primary auditory cortexneuron 2007 53 279292abstract full text full text pdf pubmed scopus 496 google scholarb soundinduced crossmodal phase locking alphaband oscillations human occipital cortex visual cortex excitability left phase dynamics eeg alpha frequency posterior recording sites response brief sound incidence preferred phase 100 ms postsound 0 300 ms sound onset eeg alphaphase dynamics correlated right soundinduced cycling visual cortex excitability first 300 ms sound onset tested phosphene perception rate response single occipital transcranial magnetic stimulation pulses findings illustrate cocycling perception underlying perceptually relevant oscillatory activity identical frequency alpharange around 10 hz adapted romei et al 2012romei v gross j thut g sounds reset rhythms visual cortex corresponding human visual perceptioncurr biol 2012 22 807813abstract full text full text pdf pubmed scopus 93 google scholarboth b support notion sensory input reset phase ongoing oscillations cortical areas specialized process another modality thereby facilitate processing certain periodic intervals suppress processing intervals mechanism crossmodal input reset oscillations enhance processing specifically times relevant input predictedview large image figure viewerdownload hires image download ppt effect somatosensoryauditory stimulus onset asynchrony soa bimodal response left color map shows eventrelated current source density csd response site large current sink supragranular layers macaque area ai different somatosensoryauditory soas csd index net synaptic responses transmembrane currents lead action potentials indexed concomitant multiunit activity mua signal local neuronal ensemble increasing soas mapped axis top bottom 0 top corresponding simultaneous auditorysomatosensory stimulation au bottom represents auditory alone condition red dotted lines denote 2060 ms time interval averaged csd mua single trials quantification right represent mean csd mua amplitude values x axis 2060 ms auditory poststimulus time interval error bars show se different somatosensoryauditory soas axis stars denote number experiments total six given soa bimodal response amplitude significantly different auditory pea