4 Sulfaro et al. clamped to some predetermined set of input values. Drawing from modelling by Breedlove et al. (2020), the highest layer in the network was also clamped to a fixed set of input values to simulate an internally generated stimulus (i. e. a thought) during mental imagery. To simulate feedforward information pooling in the visual ven-tral stream, ascending representations were combined using a 13 × 13 square convolutional kernel such that each element of the layer above received the average of inputs from multiple elements from the layer below. Hence, from l=0 to l=4, the receptive field size of each element in each representation increased while layer dimensionality decreased. Because forming a mental image of an object first requires at some point seeing the object or its con-stituent parts, internally generated stimulus representations were formed from externally generated (e. g. retinal) stimulus inputs first processed through a single complete feedforward sweep from the lowest to the highest layer. To assess the degree to which mental imagery was able to compete with veridical sensory input, we allowed the network to evolve under three main sets of initial conditions: veridical perception without imagination, imagination without veridical perception, and imagination during veridical perception. In the first case, to simulate a scenario where externally generated visual input is present without competition, only the bottom layer was clamped to a stimulus representation (veridical perception without imagination). Second, to simulate a scenario where an internally generated stimulus is present without competition, the top layer was clamped to an imagined stimulus representation (imagination without veridical perception). The ascending input from the base layer was downweighted to zero in this scenario to enforce a state of sensory disconnection. Third, in what we con-sider a more realistic mental imagery scenario, both the top and bottom layers were simultaneously clamped to different stimu-lus representations to simulate mental imagery occurring in the presence of competing sensory input (imagination during veridi-cal perception). All intervening or non-input layers were initialized with white noise. In each scenario, we measured the steady-state contribution of top-down representations relative to bottom-up representations at each layer. This was achieved by taking the average value of all elements in each layer for the case where the initial low-level or high-level input representations were set homogenously to zeros or ones, respectively (Fig. 2d-f). Each scenario was also replicated using images as inputs with each element of a representation cor-responding to one pixel in the image such that the collective pixel values of each image equate to a simulated neural activity pat-tern (Fig. 2a-c). A 256 × 256 greyscale image of a camel was used for sensory input, and a 256 × 256 greyscale image of a hat was used for imagined input after conversion to a reduced dimensionality of 208 × 208 via feedforward pooling. Both were sourced from the BOSS image database (Brodeur et al. 2010 ). The similarity between layer activity patterns and the original imagined or veridical stim-ulus patterns can be quantified by calculating the absolute mean pixel difference between the images. We also separately explored how varying the ascending and descending weighting schemes affected competition between simulated imagined and veridical stimulus representations (Fig. 3). For each scenario in Fig. 3, all mid-layer feedforward or feedback weightings were ≤1 with the ratio specified in the figure for the duration of the simulation. Model interpretation Our model can be interpreted as analogous to a simplified model of the human visual processing hierarchy, with a basal layer encoding externally generated retinal input and an apical layer housing more abstract internally generated stimulus represen-tations encoded by neural structures towards the parietal and temporal lobes ( Binder 2016 ). To interpret simulation outcomes phenomenologically, we considered that a single perceptual expe-rience can contain a distribution of information, spanning from highly sensory (modal) in nature to highly abstract (amodal). We assume that each representational layer in the visual hier-archy contributes information to the construction of conscious perceptual experience if it receives both feedforward and feed-back input. We also assume that the type of information provided by each layer aligns with the layer's dimensionality such that high-dimensional representations contribute most to the sensory aspects of the experience, while low-dimensional representations contribute most to the non-sensory aspects of the experience. Hence, low-level representations would be most influential in determining the literal image content of a visual experience (e. g. the exact location, orientation, and contrast of high-spatial fre-quency edges), while high-level representations would contribute most to the abstract understanding of what is being thought about or observed (e. g. knowing whether the edges in a scene form a hat or a camel), with a spectrum of quasi-sensory or quasi-abstract features in between. Crucially, this implies that a perceptual experience gains sensory or abstract properties depending on how well neural regions with the corresponding high-or low-dimensional struc-ture, respectively, are recruited during that experience. Therefore, the further an internally generated stimulus representation is able to spread towards the primary visual cortex, for instance, the more it may be experienced with properties approaching that of an actual visual image (i. e. a true mental image) rather than something pondered without sensory qualia (i. e. a non-sensory thought). This approach can therefore maintain consistency with traditional accounts implicating the primary visual cortex in men-tal imagery ( Kosslyn and Thompson 2003 ; Kosslyn 2005 ; Pearson and Kosslyn 2015; Pearson 2019). Note that, as hallucinatory visual experience is possible with-out corresponding retinal activity, we consider the lowest level in the hierarchy purely as a non-conscious input layer to the rest of the network. We also note that while there is compelling evi-dence that some high-level representations are stored in a largely amodal form (e. g. Popham et al. 2021), they are not necessar-ily completely non-sensory. Late-stage object representations can still contain some degree of sensory information (e. g. Di Carlo and Cox 2007), albeit in a considerably more low-dimensional form relative to primary sensory areas. Our model only requires higher-level representations to be less sensory than low-level representations rather than completely non-sensory. Results Perception without competition T wo scenarios were simulated where competition was not present: veridical perception without imagination and imagination with-out veridical perception. Initiation of veridical perception was simulated by allowing the lowest representational layer to have a non-zero contribution to the layer above. Imagination was sim-ulated by preventing the highest layer from being affected by feedforward inputs below such that it provided a constant source of top-down information to the system. In both cases, only one fixed source of perceptual information was present within the network, either an externally or internally generated stimulus rep-resentation. An example simulation is shown in Fig. 2 using an Downloaded from https://academic. oup. com/nc/article/2023/1/niad018/7248966 by guest on 25 August 2024 