summaries: Motivated by the literature showing that the primate visual system has two pathways ("ventral" and "dorsal") with distinct functional specializations, the authors try to build a model of a visual system that also has representationally and functionally distinct regions. They turn to calcium imaging data from mouse visual cortex, which allows them to compare learned representations of ANN to the neural representations found in distinct cortical areas of animals shown the same images. (There aren't publicly available datasets with simultaneous recordings from ventral and dorsal areas in primates, as far as I know.) The authors find that a two-pathway 3D ResNet model trained on the contrastive predictive coding (CPC) task and a dataset of natural videos has layers that overall explain more of the mouse calcium responses in more cortical areas than several baseline models, including the same architecture trained on the same dataset but with a supervised task. In addition, the two pathways in the self-supervised model transfer better to different tasks (object recognition and motion discrimination), which is reminiscent of some of the functional specialization between primate ventral and dorsal pathways.

summaries: The authors show that a two-stream 3d ResNet architecture trained using a contrastive predictive coding (CPC) objective develops a representation that resembles the ventral/dorsal stream split in visual cortex. Although I do not buy into all claims of the paper, I am generally very supportive of it. The authors make their main point and the paper is well written. With some adjustments of the claims it will be a great paper.   ### Update after discussion period  Given the results from the control experiments reported by the authors, I support publication of the paper and increased my score to 7. I do ask the authors to include those results in the final version of the paper.

summaries: The authors address the challenge of training a single deep neural network, using a single loss function, which captures the specialized neural responses in both the ventral (“what”) and dorsal (“where”) pathways of mouse visual cortex. The authors demonstrate that a deep neural network comprised of two parallel convolutional hierarchies, trained with a single self-supervised predictive objective, captures neural responses in the ventral and dorsal streams better than a neural network comprised of a single convolutional hierarchy (which captures only the ventral stream), and better than either architecture trained with a supervised objective. Furthermore, by evaluating their two-pathway model on “what” and “where” downstream tasks —object categorization and motion detection— the authors show that their model has learned functionally specialized representations in each of its two pathways.

summaries: The paper uses contrastive predictive coding to train deep networks on UCF101 videos. It then uses representational similarity to compare their representations to those of mouse calcium-imaging data from the Allen Brain Observatory. The key result is that a two-stream network is a good match to mouse visual cortex data. One of the (structurally identical) streams develops a representation that is a good match to dorsal mouse areas, and the other matches ventral mouse areas. Deeper areas correspond better to areas higher in the mouse visual hierarchy. Furthermore, the ventral stream of the model better supports transfer to CIFAR-10, while the dorsal stream better supports discrimination of random-dot motion direction.

abstractText: The visual system of mammals is comprised of parallel, hierarchical specialized pathways. Different pathways are specialized in so far as they use representations that are more suitable for supporting specific downstream behaviours. In particular, the clearest example is the specialization of the ventral (“what”) and dorsal (“where”) pathways of the visual cortex. These two pathways support behaviours related to visual recognition and movement, respectively. To-date, deep neural networks have mostly been used as models of the ventral, recognition pathway. However, it is unknown whether both pathways can be modelled with a single deep ANN. Here, we ask whether a single model with a single loss function can capture the properties of both the ventral and the dorsal pathways. We explore this question using data from mice, who like other mammals, have specialized pathways that appear to support recognition and movement behaviours. We show that when we train a deep neural network architecture with two parallel pathways using a self-supervised predictive loss function, we can outperform other models in fitting mouse visual cortex. Moreover, we can model both the dorsal and ventral pathways. These results demonstrate that a self-supervised predictive learning approach applied to parallel pathway architectures can account for some of the functional specialization seen in mammalian visual systems.

authors:
Pouya Bashivan
Kohitij Kar
James J DiCarlo

title: Neural population control via deep image synthesis

authors:
Doris I Braun
Duane K Boman
John R Hotson

title: Anticipatory smooth eye movements and predictive pursuit after unilateral lesions in human brain

venue: Experimental brain research,

authors:
Kenneth H Britten
William T Newsome
Michael N Shadlen
Simona Celebrini
J Anthony Movshon

title: A relationship between behavioral choice and the visual responses of neurons in macaque mt

venue: Visual neuroscience,

authors:
Kenneth H Britten
Michael N Shadlen
William T Newsome
J Anthony Movshon

title: The analysis of visual motion: a comparison of neuronal and psychophysical performance

venue: Journal of Neuroscience,

authors:
Santiago A Cadena
Fabian H Sinz
Taliah Muhammad
Emmanouil Froudarakis
Erick Cobos
Edgar Y Walker
Jake Reimer
Matthias Bethge
Andreas Tolias
Alexander S Ecker

title: How well do deep neural networks trained on object recognition characterize the mouse visual system? 2019

authors:
Charles F Cadieu
Bruno A Olshausen

title: Learning intermediate-level representations of form and motion from natural movies

venue: Neural computation,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

venue: In International conference on machine learning,

title: Whatever next? predictive brains, situated agents, and the future of cognitive science

venue: Behavioral and brain sciences,

authors:
Saskia EJ de Vries
Jerome A Lecoq
Michael A Buice
Peter A Groblewski
Gabriel K Ocker
Michael Oliver
David Feng
Nicholas Cain
Peter Ledochowitsch
Daniel Millman

title: A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex

venue: Nature neuroscience,

authors:
Katrien Denys
Wim Vanduffel
Denis Fize
Koen Nelissen
Hendrik Peuskens
David Van Essen
Guy A Orban

title: The processing of visual shape in the cerebral cortex of human and nonhuman primates: a functional magnetic resonance imaging study

authors:
Barry J Devereux
Alex Clarke
Andreas Marouchos
Lorraine K Tyler

title: Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects

authors:
Jörn Diedrichsen
Eva Berlot
Marieke Mur
Heiko H Schütt
Mahdiyar Shahbazi
Nikolaus Kriegeskorte

title: Comparing representational geometries using whitened unbiased-distance-matrix similarity

venue: arXiv preprint arXiv:2007.02789,

authors:
Christoph Feichtenhofer
Haoqi Fan
Jitendra Malik
Kaiming He

title: Slowfast networks for video recognition

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Christoph Feichtenhofer
Haoqi Fan
Bo Xiong
Ross Girshick
Kaiming He

title: A large-scale study on unsupervised spatiotemporal representation learning

venue: arXiv preprint arXiv:2104.14558,

authors:
Daniel J Felleman
David C Van Essen

title: Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex

authors:
Aris Fiser
David Mahringer
Hassana K Oyibo
Anders V Petersen
Marcus Leinweber
Georg B Keller

title: Experience-dependent spatial expectations in mouse visual cortex

authors:
Emmanouil Froudarakis
Paul G Fahey
Jacob Reimer
Stelios M Smirnakis
Edward J Tehovnik
Andreas S Tolias

title: The visual cortex in context

venue: Annual review of vision science,

authors:
Colleen J Gillon
Jason E Pina
Jérôme A Lecoq
Ruweida Ahmed
Yazan Billeh
Shiella Caldejon
Peter Groblewski
Tim M Henley
Eric Lee
Jennifer Luviano

title: Learning from unexpected events in the neocortical microcircuit

venue: bioRxiv,

authors:
Melvyn A Goodale
A David Milner

title: Separate visual pathways for perception and action

venue: Trends in neurosciences,

authors:
Jean-Bastien Grill
Florian Strub
Florent Altché
Corentin Tallec
Pierre H Richemond
Elena Buchatskaya
Carl Doersch
Bernardo Avila Pires
Zhaohan Daniel Guo
Mohammad Gheshlaghi Azar

title: Bootstrap your own latent: A new approach to self-supervised learning

venue: arXiv preprint arXiv:2006.07733,

authors:
Umut Güçlü
Marcel AJ van Gerven

title: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects

authors:
Tengda Han
Weidi Xie
Andrew Zisserman

title: Video representation learning by dense predictive coding

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops,

authors:
Julie A Harris
Stefan Mihalas
Karla E Hirokawa
Jennifer D Whitesell
Hannah Choi
Amy Bernard
Phillip Bohn
Shiella Caldejon
Linzy Casal
Andrew Cho

title: Hierarchical organization of cortical and thalamic

venue: connectivity. Nature,

authors:
James V Haxby
Cheryl L Grady
Barry Horwitz
Leslie G Ungerleider
Mortimer Mishkin
Richard E Carson
Peter Herscovitch
Mark B Schapiro
Stanley I Rapoport

title: Dissociation of object and spatial visual processing pathways in human extrastriate cortex

venue: Proceedings of the National Academy of Sciences,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Irina Higgins
Le Chang
Victoria Langston
Demis Hassabis
Christopher Summerfield
Doris Tsao
Matthew Botvinick

title: Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons

authors:
Andrew D Huberman
Cristopher M Niell

title: What can mice tell us about how vision works

authors:
Thomas W James
Jody Culham
G Keith Humphrey
A David Milner
Melvyn A Goodale

title: Ventral occipital lesions impair object recognition but not object-directed grasping: an fmri study

authors:
Judson P Jones
Larry A Palmer

title: An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex

venue: Journal of neurophysiology,

authors:
Georg B Keller
Tobias Bonhoeffer
Mark Hübener

title: Sensorimotor mismatch signals in primary visual cortex of the behaving mouse

authors:
Georg B Keller
Thomas D Mrsic-Flogel

title: Predictive processing: a canonical cortical

venue: computation. Neuron,

authors:
Seyed-Mahdi Khaligh-Razavi
Nikolaus Kriegeskorte

title: Deep supervised, but not unsupervised, models may explain it cortical representation

venue: PLoS computational biology,

authors:
Tim C Kietzmann
Courtney J Spoerer
Lynn KA Sörensen
Radoslaw M Cichy
Olaf Hauk
Nikolaus Kriegeskorte

title: Recurrence is required to capture the representational dynamics of the human visual system

authors:
Talia Konkle
George Alvarez

title: Deepnets do not need category supervision to predict visual system responses to objects

venue: Journal of Vision,

authors:
Eileen Kowler
Jason F Rubinstein
Elio M Santos
Jie Wang

title: Predictive smooth pursuit eye movements

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Marie-Eve Laramée
Denis Boire

title: Visual cortical areas of the mouse: comparison of parcellation and network structure with primates

venue: Frontiers in neural circuits,

authors:
William Lotter
Gabriel Kreiman
David Cox

title: Deep predictive coding networks for video prediction and unsupervised learning

venue: arXiv preprint arXiv:1605.08104,

authors:
James H Marshel
Marina E Garrett
Ian Nauhaus
Edward M Callaway

title: Functional specialization of seven mouse visual cortical

venue: areas. Neuron,

authors:
AJ McDonald
F Mascagni

title: Cortico-cortical and cortico-amygdaloid projections of the rat occipital cortex: a phaseolus vulgaris leucoagglutinin study

authors:
Johannes Mehrer
Courtney J Spoerer
Nikolaus Kriegeskorte
Tim C Kietzmann

title: Individual differences among deep neural network models

venue: Nature communications,

authors:
Mortimer Mishkin
Leslie G Ungerleider
Kathleen A Macko

title: Object vision and spatial vision: two cortical pathways

authors:
Thao Nguyen
Maithra Raghu
Simon Kornblith

title: Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth

authors:
Cristopher M Niell
Massimo Scanziani

title: How cortical circuits implement cortical computations: Mouse visual cortex as a model

venue: Annual Review of Neuroscience,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Christopher C Pack
Richard T Born

title: Temporal dynamics of a neural solution to the aperture problem in visual area mt of macaque

venue: brain. Nature,

authors:
Tatiana Pasternak
Kris M Horn
John HR Maunsell

title: Deficits in speed discrimination following lesions of the lateral suprasylvian cortex in the cat

authors:
Tatiana Pasternak
William H Merigan

title: Motion perception following lesions of the superior temporal sulcus in the monkey

venue: Cerebral Cortex,

authors:
Carlos R Ponce
Will Xiao
Peter F Schade
Till S Hartmann
Gabriel Kreiman
Margaret S Livingstone

title: Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal

venue: preferences. Cell,

authors:
Blake A Richards
Timothy P Lillicrap
Philippe Beaudoin
Yoshua Bengio
Rafal Bogacz
Amelia Christensen
Claudia Clopath
Rui Ponte Costa
Archy de Berker
Surya Ganguli

title: A deep learning framework for neuroscience

authors:
Reuben Rideaux
Andrew E Welchman

title: But still it moves: static image statistics underlie how we see motion

authors:
Peter H Schiller
Nikos K Logothetis

title: The color-opponent and broad-band channels of the primate visual system

authors:
Jianghong Shi
Eric Shea-Brown
Michael A Buice

title: Comparison against task driven artificial neural networks reveals functional organization of mouse visual cortex

venue: arXiv preprint arXiv:1911.07986,

authors:
Kevin K Sit
Michael J Goard

title: Distributed and retinotopically asymmetric processing of coherent motion in mouse visual cortex

authors:
Darragh Smyth
Ben Willmore
Gary E Baker
Ian D Thompson
David J Tolhurst

title: The receptive-field organization of simple cells in primary visual cortex of ferrets under natural scene stimulation

authors:
Khurram Soomro
Amir Roshan Zamir
Mubarak Shah

title: Ucf101: A dataset of 101 human actions classes from videos in the wild

venue: arXiv preprint arXiv:1212.0402,

authors:
Katherine R Storrs
Tim C Kietzmann
Alexander Walther
Johannes Mehrer
Nikolaus Kriegeskorte

title: Diverse deep neural networks all predict human it well, after training and fitting

authors:
Quanxin Wang
Enquan Gao
Andreas Burkhalter

title: Gateways of ventral and dorsal streams in mouse visual cortex

authors:
Quanxin Wang
Olaf Sporns
Andreas Burkhalter

title: Network analysis of corticocortical connections reveals ventral and dorsal processing streams in mouse visual cortex

authors:
Scott NJ Watamaniuk
Robert Sekuler

title: Temporal and spatial integration in dynamic randomdot stimuli

venue: Vision research,

authors:
Laurenz Wiskott
Terrence J Sejnowski

title: Slow feature analysis: Unsupervised learning of invariances

authors:
Daniel LK Yamins
Ha Hong
Charles F Cadieu
Ethan A Solomon
Darren Seibert
James J DiCarlo

title: Performance-optimized hierarchical models predict neural responses in higher visual cortex

venue: Proceedings of the national academy of sciences,

authors:
Chengxu Zhuang
Siming Yan
Aran Nayebi
Martin Schrimpf
Michael C Frank
James J DiCarlo
Daniel LK Yamins

title: Unsupervised neural network models of the ventral visual stream

heading: 1 Introduction

text:
In the mammalian visual cortex information is processed in a hierarchical manner using two specialized pathways [15, 52]: the ventral, or “where” pathway, and the dorsal, or “what” pathway. These two pathways are specialized for visual recognition and movement, respectively [42, 24, 19, 58, 59, 17]. For example, damage to the ventral pathway may impair object recognition, whereas damage to the dorsal pathway may impair motion perception [64].
Deep artificial neural networks (ANNs) trained in a supervised manner on object categorization have been successful at matching the representations of the ventral visual stream [62, 57, 32]. They have
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
been shown to develop representations that map onto the ventral hierarchy, and which can be used to predict [62] and control [1, 49] neural activity in the ventral pathway. However, when we look at the other principal visual pathway in the mammalian brain, i.e. the dorsal pathway, the situation is different. Very few studies have examined the ability of deep ANNs to develop representations that match the dorsal hierarchy (though see the following fMRI study: [21]). Moreover, to the best of our knowledge, no studies have demonstrated both ventral-like and dorsal-like representations in a single network.
This lack of deep ANN models that capture both ventral and dorsal pathways leads naturally to an important question: under what training conditions would specialized ventral-like and dorsal-like pathways emerge in a deep ANN? Would a second loss function be required to obtain matches to dorsal pathways, or is there a single loss function that could induce both types of representations?
One promising set of candidates are predictive self-supervised loss functions [45, 22, 38]. Recent work has shown that self-supervised learning can produce similar results to supervised learning for the ventral pathway [63, 34, 26]. Moreover, there is a large body of work showing that mammals possess predictive processing mechanisms in their cortex [31, 8, 30, 18, 16], including in the dorsal pathway [2, 35], which suggests that a predictive form of self-supervised learning could potentially lead to the emergence of both ventral and dorsal-like representations.
Addressing this question requires recordings from different ventral and dorsal visual areas in the brain. Here, we explore these issues using publicly available data from the Allen Brain Observatory [9], which provides recordings from a large number of areas in mouse visual cortex. We examine the ability of a self-supervised predictive loss function (contrastive predictive coding [45, 22]) to induce representations that match mouse visual cortex. When we train a network with a single pathway, we find that it possesses more ventral-like representations. However, when we train a network with two parallel pathways we find that the predictive loss function induces distinct representations that map onto the ventral/dorsal division. This allows the network to better support both object categorization and motion recognition downstream tasks via the respective specialized pathways. In contrast, supervised training with an action categorization task only leads to matches to the ventral pathway, and not the dorsal pathway.
Altogether, this work demonstrates that the two specialized pathways of visual cortex can be modelled with the same ANN if a self-supervised predictive loss function is applied to an architecture with parallel pathways. This suggests that self-supervised predictive loss functions may hold great promise for explaining the functional properties of mammalian visual cortex.

heading: 2 Background and related work

text:
Self-supervised ANN models of the ventral visual stream Recently, [63] and [34] showed that the representations learned by self-supervised models trained on static images produce good matches to the ventral pathway. Our work builds on this by exploring the potential for self-supervised learning to also explain the dorsal pathway.
ANN models of the mouse visual system Mice have become a common animal model in visual neuroscience due to the sophisticated array of experimental tools [44, 27]. As such, [53] compared the responses of different areas in mouse visual cortex with the representations of a VGG16 trained on ImageNet. In this paper, we show that self-supervised learning can produce better fits to both ventral and dorsal areas than supervised learning.
ANN models of the dorsal pathway The MotionNet ANN model [51], is a feedforward ANN trained to predict the motion direction of segments of natural images, which was inspired by the role of the dorsal pathway in motion perception. In [6], they show that learning both ventral and dorsal-like representations in a single ANN with two pathways is possible if one forces the two pathways to process the phase and amplitude of a complex decomposition of the stimuli separately. In [21], they show that training a deep ANN on supervised action recognition can induce some match to dorsal pathway fMRI recordings. In this study, we show that with prediction as the learning objective and an architecture that has two parallel pathways, both ventral-like and dorsal-like representations can be learned.

heading: 3 Methods

heading: 3.1 Datasets

text: We use the Allen Brain Observatory open 2-photon calcium imaging dataset for the experiments in this study. We select subsets of the dataset based on brain area, recording depth, and visual stimuli used. Recordings from five areas of mouse visual cortex are used (VISp, VISlm, VISal, VISpm, VISam; Figure 1a). We only exclude one area from our analyses (VISrl) because it is a multi-sensory area, and visual stimuli alone do not drive it well [9]. We use recordings from cortical depths of 175-250µm (which corresponds to cortical layers 2-3) as these are the recordings with the largest number of neurons. When selecting visual stimuli, we use recordings elicited by the presentation of natural movies, because unlike static images, movies can elicit clear responses in both ventral and dorsal areas [9]. Thus, we only use those parts of the dataset in which natural movies (30 seconds long) were presented as visual stimuli. More details about the dataset can be found in [9]. For training the deep ANNs, we use the UCF101 dataset [56] (see section A.4 of supplementary materials).

heading: 3.2 Analysis techniques

text:
Representational Similarity Analysis (RSA) To measure the representational similarities between real brains and ANNs we use the RSA method. RSA has been commonly used in both the neuroscience [11, 33] and deep learning literature [41, 43]. The details of RSA can be found in those citations, but we will summarize it briefly here. First, we create response matrices R ∈ IRN×M for every brain area and every layer of our ANNs, where N is the number of neurons and M is the number of video blocks, with each block comprising 15 frames of the video. Element ij of the response matrix represents the response of the ith neuron to the jth video sequence. We then use Pearson correlation to calculate the similarity of every pair of columns (e.g. kth and lth columns) in matrix R, and form the M by M Representation Similarity Matrix (RSM ∈ IRM×M ), in which every element (kl) quantifies the similarity of the responses to the kth and lth videos blocks. Thus, the RSMs describe the representation space in a network, be it a brain area or an ANN layer. Given two RSMs, we then use Kendall’s τ between the vectorized RSMs to quantify the similarity between the two representations. It is important to note that measurement noise can potentially induce bias in the RSM estimations, but since the variance of measurement noise is not expected to be different across different video blocks, using Kendall’s τ rank correlation should cancel the bias in our RSM estimations (see [12] for more details). As an additional sanity check, we use RSA to compare the representations between mice. If RSA is identifying salient representational geometries, then the RSMs between different areas should be lower than the RSMs for the same areas. Indeed, as shown in Figure 1b, the representational similarity is highest for the same regions across animals (i.e. the diagonal values are larger). These diagonal values also represent the noise ceiling for these areas (see section A.1). Throughout the paper (with the exception of Figure 1b), RSM similarities are reported as the percentage of the noise ceiling.
Identification of brain regions in the hierarchy The hierarchical organization of mouse visual areas can be inferred based on the anatomical and functional properties of each brain region. We adopted an approximation of hierarchical indices as reported in [23]. The relative hierarchical placing of the visual areas included in this study are shown in Figure 1c.
Identification of brain regions into ventral and dorsal streams We group the brain regions into two sets, i.e. ventral and dorsal areas. Although the ventral/dorsal specialization of visual pathways in mice is not as clear and well understood as it is in primates, many anatomical and physiological studies do suggest that mice also possess such specialized pathways [40, 58, 39, 37]. According to previous anatomical and physiological studies [17, 58], VISlm and VISam can be considered as the most ventral and dorsal areas of mouse visual cortex, respectively. We then use the similarity of representations between other areas and VISlm and VISam to estimate a ventral score (V-score = rAlm/(r A am + r A lm)) and a dorsal score (D-score = r A am/(r A am + r A lm)) for each area. rAam is the representational similarity between area A and VISam, and r A lm is the representational similarity between area A and VISlm. The D-score and V-score values for the five visual areas are plotted in Figure 1d. Based on the D-score and V-score values, we grouped VISlm, VISp, and VISpm as more ventral and VISal, and VISam as more dorsal in our analysis. This grouping is in keeping
with a recent study that showed VISam and VISal are the first and second most responsive areas to motion stimuli: an important characteristic of dorsal areas [54].

heading: 3.3 Model

text:
Self-supervised predictive learning We used Contrastive Predictive Coding (CPC) for selfsupervised learning, which was developed for modeling sequential data, including video datasets [45]. The loss function relies on predicting the future latent representations of a video sequence, given its present and past representations. See Figure S1 for a schematic of the model, and A.2 for more details regarding the CPC loss function.
ANN Architecture All the ANN backbones used in our study are variants of the ResNet architecture, similar to the ones used in [13]. Our ResNet architectures have either one or two pathways. The one-pathway ResNet (ResNet-1p) is a regular 3D ResNet (Figure 2a). The two-pathway ResNet (ResNet-2p) is composed of two parallel ResNet branches, which split after a single convolutional layer, and merge after their final layers through concatenating their outputs along the channel dimension (Figure 3a). Both pathways of ResNet-2p receive a copy of the first convolutional layer output, and each has 10 Res-blocks. To keep the total number of output channels the same in both architectures, each pathway in ResNet-2p has half the number of channels of the single pathway in ResNet-1p. For the ANNs trained on object categorization, we use ResNet-18 pretrained on the ImageNet database [25]. Our 3D ResNet architectures are summarized in Table S1.
Baseline models In terms of alignment with brain representations of video sequences, we compare the CPC trained ANNs with four other models: (1) a simple model based on Gabor filters, (2) a randomized deep ANN, (3) a ResNet-18 trained on ImageNet, and (4) 3D ResNets trained on action recognition in a supervised manner. See section A.3 of supplementary materials for more details.
Training We use the backpropagation algorithm and Adam optimizer. CPC is trained with a batch size of 40 samples, a learning rate of 10−3, and 100 epochs. Supervised action recognition is trained with a batch size of 256 samples, a learning rate of 5× 10−4, and 300 epochs. All the models are implemented with PyTorch 1.7 and trained on RTX8000 NVIDIA GPUs.
Downstream tasks In addition to comparisons with dorsal and ventral representations in mouse brain, we examine the two pathways of our trained ResNet-2p on two downstream tasks: object categorization and motion discrimination, which are supported in the real brain by the ventral and dorsal pathways, respectively. See section A.5 of supplementary materials for more details.

heading: 4 Results

heading: 4.1 CPC with a single pathway architecture produces better matches to mouse ventral stream

text:
We first compare the representations learned with CPC on a single pathway architecture (Fig. 2a) with the three ventral areas of mouse visual cortex (VISlm, VISp, and VISpm; based on D-scores/V-scores in Figure 1d). In Figure 2b (top), the similarity of RSMs between different layers of the ANN and these three areas are shown. Comparing the maximum representational similarities between models in Figure 2c, we can see that, for the three ventral areas, CPC shows a higher maximum similarity compared to the other models. Compared to the baseline models (untrained ANN and Gabors), the ANN trained with object categorization has higher representational similarity to VISlm and VISp, the two most ventral areas (see Figure 1d), which is consistent with the suggested role of ventral areas in form and shape representation [10, 17]. This is in contrast with a previous study [5] that could not find any significant difference between ANNs trained on object categorization and untrained ANNs in modeling mouse visual cortex. There are several sources of variability that could explain this contradiction (e.g. architecture, datasets, etc.), but an important possibility is that the two studies used different visual stimuli, namely, natural videos in our study vs static natural images in [5]. Natural videos are better suited for eliciting sufficiently strong responses that can distinguish between the representations of the object categorization-trained and randomly initialized ANNs [9].
As noted, and as can be seen in Figure 2b, the maximum similarity happens in different hierarchical levels of the CPC-trained ANN for each area. We quantify the hierarchy index for every area by dividing the layer number with the highest similarity by the maximum number of layers. Based on this measure, the areas at the very top and bottom of the visual hierarchy have a hierarchy index of 1 and 0, respectively. The hierarchy index for the three areas are shown in Figure 2d. In accordance with the anatomical and functional data (Figure 1c), VISp has a lower hierarchical index (0.41±0.15) than VISlm (0.61 ± 0.08) and VISpm (0.55 ± 0.08). It should be noted, that despite the fact that anatomical/functional hierarchy indices position VISpm higher in the hierarchy than VISlm, our model-based measure of hierarchy places the two areas at around a similar level, with VISlm having a marginally higher value than VISpm. We speculate that this may be a result of the fact that VISpm has noisier representations, and therefore a lower noise ceiling (Fig. 1b). This may be due to VISpm actually being more multi-modal than purely visual.

heading: 4.2 CPC on a single pathway does not match mouse dorsal stream

text: We compare the similarity of RSMs between ResNet-1p trained with CPC (Fig. 2a) and the dorsal areas (VISam and VISal; based on D-scores/V-scores in Figure 1d). As shown in Figure 2b (bottom), the ANN trained with CPC just passes the pixel-level representations in its early layers (the first gray circle in the plots), and then the similarity values continuously decrease for deeper layers. Notably, for the most dorsal area (VISam), the maximum similarity of RSMs does not go above the untrained model (Figure 2c). For VISal, even though CPC shows some improvement compared to an untrained ANN, the performance is much lower than for the ventral areas, as shown in the previous section (Fig. 2c). These findings show that the representations learned by CPC and the ResNet-1p architecture are more ventral-like, and do not easily explain dorsal area representations (see supplementary section E). Similarly, an object categorization trained ANN has a very low similarity with the dorsal areas (even lower than the untrained ANN for VISam) indicating that object categorization cannot be considered as an appropriate loss function for the dorsal pathway, which is consistent with our current understanding in neuroscience.

heading: 4.3 An architecture with parallel pathways trained with CPC can model both ventral and dorsal areas

text:
CPC’s tendency to learn ventral-like representations, as seen in the previous section, could be due to a limitation of the backbone architecture. Predicting the next frame of a video with a contrastive loss requires learning invariances to every kind of transformation (or augmentation) that could happen from one frame to the next. For example, representing objects’ shapes requires invariance to objects’ motion, and representing objects’ motion requires invariance to objects’ shape. As suggested in [6], these two types of invariances often underlie the distinction between the dorsal and ventral representations in the brain. Therefore, one possibility is that ventral and dorsal-like representations
compete for resources in the network, and ultimately, the ventral-like representations win out, possibly because they provide a greater overall reduction in loss. If this hypothesis is true, then a network with two separate pathways may be able to reduce the competition by assigning one pathway to be more ventral-like, and one more dorsal-like.
To test this hypothesis, we use the simplest extension of ResNet-1p: ResNet-2p, a ResNet architecture that is composed of two identical, parallel ResNets which split after the first layer and merge after their final layers (Fig. 3a; see also section 3.3). Each pathway of ResNet-2p has half the number of channels of ResNet-1p, keeping the total number of channels per layer equal between the two architectures. We choose this ResNet-2p architecture as it shares all of the features of ResNet-1p, but with two separate pathways that could potentially be assigned specialized functions. We then check whether ResNet-2p can learn both ventral- and dorsal-like representations. We compare the representations of all the layers of the two pathways in ResNet-2p (blue and red in the schematic in Fig. 3a) with all the visual areas. The results show that the representations along one of the pathways (the blue pathway in Fig. 3a-b, top) are more ventral-like, and the representations along the other pathway (the red pathway in Fig. 3a-b, bottom) are more dorsal-like. Therefore, the two pathways together can model both ventral and dorsal areas of mouse visual cortex. Compared to an untrained ANN, both ResNet-1p and ResNet-2p achieve high RSM similarity values for ventral areas (Figure 3c), with ResNet-2p showing a slight decrease in VISlm and VISpm compared to ResNet-1p. However, unlike ResNet-1p, which fails to model the dorsal areas, ResNet-2p outperforms ResNet-1p and the untrained ANN for area VISam by a wide margin (Fig. 3c). For VISal, maximum RSM similarity values for ResNet-1p and ResNet-2p are around the same level (Fig. 3c), though the similarity values of the red pathway representations do not drop as much throughout the network as the blue pathway representations do, indicating the general similarity of the red pathway to the dorsal areas. Examination of the RSM similarity between the two pathways shows that the representational geometries are quite different (Fig. S2). Moreover, when we compare the RSMs of the two pathways with those from ResNet-1p, we can see that ResNet-1p has representations that are a better match
to the ventral-like pathway from ResNet-2p (Fig. S3). This supports the idea that, in the single pathway model, there is a competition between the two forms of representation that favours ventrallike functions, and which leads to specialized functions in the two-pathway architecture. Using architectures with more than two parallel pathways also does not improve representational similarities (Fig. S5).
The hierarchy index values for the ventral and dorsal areas are shown in Figure 3d. The hierarchy index for every area is calculated based on the model pathway that aligns best with that area in terms of representation similarity (blue pathway for VISlm, VISp, VISpm, and red pathway for VISam, VISal). The hierarchy indices are shown separately for the ventral and dorsal pathways. Similar to the results with ResNet-1p for the ventral pathway, VISp has a lower hierarchy index (0.25± 0) than VISlm (0.31± 0.05) and VISpm (0.29± 0.07). For the dorsal pathway, VISam has a larger hierarchy index than VISal (VISam: 0.82 ± 0.16 vs. VISal: 0.19 ± 0.15) which is also consistent with the anatomical/functional hierarchy index (Fig. 1c).
In terms of the predictive loss function, the performances of ResNet-1p and ResNet-2p are not significantly different (top-3 accuracy for ResNet-1p: 94.64 (0.68) and ResNet-2p: 93.472 (0.83)). However, it is worth noting that ResNet-2p has, in total, fewer parameters than ResNet-1p (ResNet-1p: 435k vs ResNet-2p: 285k). Therefore, considering the lower capacity of ResNet-2p and its similar predictive performance to ResNet-1p, we can conclude that the inductive bias of parallel architecture could help in predictive processing.

heading: 4.4 Supervised learning of action recognition with parallel-pathways is not sufficient for dorsal match

text:
The inductive bias of an architecture with parallel pathways combined with the spatiotemporal dynamics of the video data could be enough to trigger the emergence of both ventral and dorsal-like representations, as defined in the previous section, regardless of the loss function used. To understand the role of the loss function, we trained the same ResNet-1p and ResNet-2p backbones with a
supervised action classification loss on the UCF101 dataset (Fig. 4a). Based on the representation similarity plots in Figure 4b, both blue and red pathways seem to learn similar representations that are more ventral-like (also see Fig. S2). The maximum RSM similarity values in Figure 4c show that both ResNet-1p and ResNet-2p reach similarity values higher than the untrained ANN for ventral-like areas, but for the dorsal areas, and specifically VISam, neither architecture achieves good representational similarity. The low performance of the action classification models here could not be due to the low spatial resolution of the training videos as our comparisons with ANNs pretrianed with higher spatial resolution videos also show similar results (see supplementary section F).
The hierarchy indices calculated using the ANN trained with an action classification objective (Fig. 4d), reproduce the CPC results (Fig. 3d and 2d) for ventral-like areas, and roughly match the anatomical/functional hierarchy index. However, the action classification model fails to predict the hierarchical organization of the dorsal areas, which is to be expected given the model’s poor alignment with these areas. Overall, these findings demonstrate the importance of the CPC loss function for learning both ventral and dorsal-like representations across the ResNet-2p architecture.

heading: 4.5 Functional specialization of the ventral and dorsal-like pathways in a CPC trained model

text:
The ventral pathway is responsible for object and scene-based tasks [28], while the dorsal pathway is responsible for motion-based tasks [3, 47, 48, 54]. Based on our knowledge of the functional specialization of the two pathways in the real brain, we run a linear evaluation on the two pathways of ResNet-2p trained with CPC on two downstream tasks: (1) object categorization (CIFAR10 dataset) and (2) motion discrimination with random dot kinematograms (RDKs) (see Fig. 5a). In vision neuroscience, RDKs have been commonly used to characterise motion representation in the dorsal pathway [4]. We use this stimulus to evaluate the CPC-trained ResNet-2p (the red pathway in Fig. 3) on motion direction discrimination (four directions: up, down, left, right). Figure 5b shows the results for the two pathways, as well as for an untrained ResNet. As expected based on the comparisons with
mouse visual areas, the ventral-like pathway is better in object categorization, while the dorsal-like pathway outperforms the ventral-like pathway in motion discrimination. Therefore, in addition to their better fits to neural data, the two pathways are indeed more ventral-like and dorsal-like according to their ability to support downstream tasks.
As previously stated, decreasing the dots coherence (increasing spatial noise) makes the randomdots task more difficult. In real brains, dorsal areas can average out noise and extract motion signals by integrating motion over both time and space [46, 60]. It is thought that this is key to the importance of dorsal areas for RDK tasks. We measured ResNet-2p ventral-like and dorsallike pathway performances for different coherence levels of random-dots (Fig. 5c). Ventral-like performance was dramatically reduced when the coherence was lowered from 100% to 50%. In contrast, the decrease in dorsal-like pathway performance was much smaller, demonstrating the dorsal-like pathway’s ability to spatially integrate motion, which is consistent with our expectations of dorsal areas. Indeed, the dorsal-like pathway can still achieve just under 40% accuracy even with 20% coherence. We also measured ventral-like and dorsal-like pathway performances for different numbers of frames of the random-dots stimuli at 50% coherence (Fig. 5d). The dorsal-like pathway’s performance improved significantly when the number of frames was increased, illustrating the ability of the dorsal-like pathway to integrate motion information over time. On the other hand, increasing the number of frames did not increase ventral-like pathway accuracy, but rather reduced it to around chance level. This shows that unlike the dorsal-like pathway, the ventral-like pathway of ResNet-2p does not integrate motion information over time.

heading: 5 Discussion

text:
In this paper, we showed that self-supervised learning with CPC can produce representations that are more analogous to mouse visual cortex than either simple models or ANNs trained in a supervised manner. Furthermore, we showed that CPC applied to an architecture that has two parallel pathways can model both the ventral and dorsal areas of mouse visual cortex. The downstream tasks of object recognition and motion discrimination also support the ventral-like and dorsal-like representations of the two pathways in the model. Our experiments with supervised training on action classification indicated that the two-pathway architecture and video dataset are necessary but not sufficient for learning both types of representations, showing an interaction between the self-supervised objective function and the architecture. This finding shows that self-supervised predictive learning is a required component of our model for obtaining both ventral- and dorsal-like representations. Our observation that supervised action classification cannot generate dorsal representations is in contradiction with a previous fMRI study [21]. The different results of the two studies boil down to the different data modalities used in the two studies: two-photon calcium imaging in our study and fMRI in [21]. Capturing the responses of the dorsal areas elicited by natural videos with high temporal dynamics requires neuronal recordings with high temporal sampling rate. Therefore, we hypothesize that some aspects of dorsal representations of movement would not be reflected in the fMRI data, which
could bias the fMRI-based analysis toward more static and lower temporal frequency features of the stimulus and neuronal responses (e.g. very slow-varying features; [61]).
Even though our results demonstrated the importance of self-supervised predictive learning for generating ventral- and dorsal-like representations, it is important to note that our conclusions are limited to the supervised loss functions that we included in our comparisons (i.e. object categorization and action recognition). We acknowledge that a different combination of more ecologically relevant supervised loss functions might be sufficient for learning ventral and dorsal representations in ANNs.
Learning representations of input data (images, videos, etc.) that are invariant to certain data augmentations (e.g. rotation, cropping, etc.) is a common goal of modern self-supervised learning methods. In models such as SimCLR [7] and BYOL [20], the augmentations are engineered for learning the most appropriate representations for downstream tasks. In CPC, however, the augmentations are inherent to the data being used. As noted above, predicting the next frame in a movie requires two different invariances: (1) invariance to motion, but selective for shape, and (2) invariance to shape, but selective for motion. Our results suggest that these two types of invariances are mutually exclusive, which can explain both the need for two separate pathways to get good matches to both ventral and dorsal areas and the inverse relationship between ventral-likeness and dorsal-likeness of the learned representation (see section D and Fig. S4 in supplementary materials). Thus, our results suggest that the functional specialization observed in the mammalian brain may be a natural consequence of a predictive objective applied to an architecture with two distinct pathways.

heading: 6 Limitations

text:
One limitation of this work is the lack of comparisons with ANNs that are trained with other predictive loss functions, such as PredNet [38]. Other self-supervised video-based learning models (for example, see [14]) that do not optimize a predictive loss function could also be compared with CPC in terms of matching the representations of mouse visual cortex. Another limitation concerns the backbone architectures that we used in this study. Different parameters of the architectures (such as the number of residual blocks, the number of layers before the split and after the merger of the two pathways in in ResNet-2p, etc.) could be searched more thoroughly to determine the optimal setting for modeling mouse visual cortex. Furthermore, training ResNet-2p with CPC on a synthetic video dataset in which the motion and shape contents of the videos can be controlled could demonstrate more directly that the two pathways of ResNet-2p learn motion-invariant shape selectivity and shape-invariant motion selectivity, respectively.
Broader Impact
ANNs can serve as a framework for understanding brains [50], as demonstrated here. This understanding would be based on finding the loss functions, architecture, and learning rules that best capture brain representations. Technologies that directly or indirectly interface with the brain, such as brain machine interfaces, can benefit from an in silico model of the brain. ANNs, if being used as such models, can facilitate designing and optimizing these technologies. However, the downside is that the ANN models of the brain are prone to the same limitations encountered by other ANNs, such as adversarial attack or implicit bias. These limitations can potentially leak into the applications in which these models would be used with human subjects.
Acknowledgments and Disclosure of Funding
We thank Iris Jianghong Shi, Michael Buice, Stefan Mihalas, Eric Shea-Brown, and Bryan Tripp for helpful discussions. We also thank Pouya Bashivan and Alex Hernandez-Garcia for their suggestions on the manuscript. This work was supported by a NSERC (Discovery Grant: RGPIN-2020-05105; Discovery Accelerator Supplement: RGPAS-2020-00031), Healthy Brains, Healthy Lives (New Investigator Award: 2b-NISU-8; Innovative Ideas Grant: 1c-II-15), and CIFAR (Canada AI Chair; Learning in Machine and Brains Fellowship). CCP was funded by a CIHR grant (MOP-115178). This work was also funded by the Canada First Research Excellence Fund (CFREF Competition 2, 2015-2016) awarded to the Healthy Brains, Healthy Lives initiative at McGill University, through the Helmholtz International BigBrain Analytics and Learning Laboratory (HIBALL).

title: The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning


summaries: (Summary) This paper proposes a generative model that can learn hierarchical topics that coherently incorporate a structured prior knowledge graph. By introducing Gaussian embeddings to an existing SawETM, a neural topic model where topics in different levels can relate each other, the authors first propose the Gaussian SawETM. Then their TopicNet based on Gaussian SawETM can learn hierarchical topics coherently to an input knowledge-graph. TopicNet is not only capable of learning hypernym-induced topics as hierarchies, but also equipped with rich interpretability.

summaries: This paper mainly proposes a new hierarchical topic discovery framework that is guided by prior (tree-structured) knowledge. The construction of the framework is broken down into two steps: Modeling semantic hierarchy with a Gaussian-embedding-based topic model (Gaussian SawETM), and injecting the prior semantic knowledge by regularizing via entailment relationship. The model is evaluated on a set of benchmarks with perplexity, topic quality, and clustering/classification protocols. Qualitative studies are also shown to demonstrate the interpretability of the model.

summaries: The paper builds on the recently proposed SawETM hierarchical topic model and combines it with a knowledge graph so that each node in the knowledge graph corresponds to a topic. They change it so that every topic is embedded with a Gaussian distribution. This enables to enforce a hierarchical ordering constraint on the topics. Experiments show that the model discovers deeper, more interpretable topics and finds better document representations.

summaries: The authors propose a topic model that aim at embedding words and topics into a latent space via Gaussian embeddings. Additional constraints in the training procedure allows to incorporate prior knowledge on the topics, so that they are able to include semantic topic relations.  The model is evaluated on a large set of public datasets and on a number of tasks. It achieves better results with respect to baselines on clustering tasks, and comparable on classification tasks. However, the main advantage is the increase interpretability of topics and topic hierarchy.

abstractText: Existing deep hierarchical topic models are able to extract semantically meaningful topics from a text corpus in an unsupervised manner and automatically organize them into a topic hierarchy. However, it is unclear how to incorporate prior belief such as knowledge graph to guide the learning of the topic hierarchy. To address this issue, we introduce TopicNet as a deep hierarchical topic model that can inject prior structural knowledge as an inductive bias to influence the learning. TopicNet represents each topic as a Gaussian-distributed embedding vector, projects the topics of all layers into a shared embedding space, and explores both the symmetric and asymmetric similarities between Gaussian embedding vectors to incorporate prior semantic hierarchies. With an auto-encoding variational inference network, the model parameters are optimized by minimizing the evidence lower bound and a regularization term via stochastic gradient descent. Experiments on widely used benchmarks show that TopicNet outperforms related deep topic models on discovering deeper interpretable topics and mining better document representations.

authors:
David M Blei
Andrew Y Ng
Michael I Jordan

title: Latent Dirichlet Allocation

venue: Journal of machine Learning research,

authors:
Mingyuan Zhou
Lauren Hannah
David Dunson
Lawrence Carin

title: Beta-negative binomial process and Poisson factor analysis

venue: In Artificial Intelligence and Statistics,

authors:
David M Blei
Thomas L Griffiths
Michael I Jordan

title: The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies

venue: Journal of the ACM (JACM),

authors:
John Paisley
Chong Wang
David M Blei
Michael I Jordan

title: Nested hierarchical Dirichlet processes

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Zhe Gan
Changyou Chen
Ricardo Henao
David Carlson
Lawrence Carin

title: Scalable deep Poisson factor analysis for topic modeling

venue: In International Conference on Machine Learning,

authors:
Mingyuan Zhou
Yulai Cong
Bo Chen

title: Augmentable gamma belief networks

venue: The Journal of Machine Learning Research,

authors:
Yulai Cong
Bo Chen
Hongwei Liu
Mingyuan Zhou

title: Deep latent Dirichlet Allocation with topic-layer-adaptive stochastic gradient riemannian mcmc

venue: arXiv preprint arXiv:1706.01724,

authors:
He Zhao
Lan Du
Wray Buntine
Mingyuan Zhou

title: Dirichlet belief networks for topic structure learning

venue: arXiv preprint arXiv:1811.00717,

authors:
Dandan Guo
Bo Chen
Hao Zhang
Mingyuan Zhou

title: Deep Poisson gamma dynamical systems

venue: arXiv preprint arXiv:1810.11209,

authors:
Chaojie Wang
Bo Chen
Mingyuan Zhou

title: Multimodal Poisson gamma belief network

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Chaojie Wang
Bo Chen
Sucheng Xiao
Mingyuan Zhou

title: Convolutional Poisson gamma belief network

venue: In International conference on machine learning,

authors:
Chaojie Wang
Hao Zhang
Bo Chen
Dongsheng Wang
Zhengjue Wang
Mingyuan Zhou

title: Deep relational topic modeling via graph Poisson gamma belief network

venue: Advances in Neural Information Processing Systems,

authors:
Jonathan Chang
Jordan Boyd-Graber
Chong Wang
Sean Gerrish
David M Blei

title: Reading tea leaves: How humans interpret topic models

venue: In Neural information processing systems,

authors:
Zhibin Duan
Dongsheng Wang
Bo Chen
Chaojie Wang
Wenchao Chen
Yewen Li
Jie Ren
Mingyuan Zhou

title: Sawtooth factorial topic embeddings guided gamma belief network

venue: In ICML 2021: International Conference on Machine Learning,

authors:
Yu Meng
Yunyi Zhang
Jiaxin Huang
Yu Zhang
Chao Zhang
Jiawei Han

title: Hierarchical topic mining via joint spherical tree and text embedding

venue: In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,

title: Wordnet: a lexical database for english

venue: Communications of the ACM,

authors:
Sumit Chopra
Raia Hadsell
Yann LeCun

title: Learning a similarity metric discriminatively, with application to face verification

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05),

authors:
Ivan Vendrov
Ryan Kiros
Sanja Fidler
Raquel Urtasun

title: Order-embeddings of images and language

venue: arXiv preprint arXiv:1511.06361,

authors:
Ben Athiwaratkun
Andrew Gordon Wilson

title: Hierarchical density order embeddings

venue: arXiv preprint arXiv:1804.09843,

authors:
X. Chen
D.P. Kingma
T. Salimans
Y. Duan
P. Dhariwal
J. Schulman
I. Sutskever
P. Abbeel

title: Variational lossy autoencoder

authors:
I.V. Serban
A. Sordoni
R. Lowe
L. Charlin
J. Pineau
A. Courville
Y. Bengio

title: A hierarchical latent variable encoder-decoder model for generating dialogues

authors:
Avd Oord
O. Vinyals
K. Kavukcuoglu

title: Neural discrete representation learning

authors:
S. Zhao
J. Song
S. Ermon

title: Towards deeper understanding of variational autoencoding models

authors:
Casper Kaae Sønderby
Tapani Raiko
Lars Maaløe
Søren Kaae Sønderby
Ole Winther

title: Ladder variational autoencoders

venue: In Advances in neural information processing systems,

authors:
Hao Zhang
Bo Chen
Dandan Guo
Mingyuan Zhou

title: WHAI: Weibull hybrid autoencoding inference for deep topic modeling

venue: In International Conference on Learning Representations,

authors:
Luke Vilnis
Andrew McCallum

title: Word representations via Gaussian embedding

venue: arXiv preprint arXiv:1412.6623,

authors:
Tony Jebara
Risi Kondor
Andrew Howard

title: Probability product kernels

authors:
Akash Srivastava
Charles Sutton

title: Autoencoding variational inference for topic models

venue: arXiv preprint arXiv:1703.01488,

authors:
Xinjie Fan
Shujian Zhang
Bo Chen
Mingyuan Zhou

title: Bayesian attention modules

venue: NeurIPS

authors:
Shujian Zhang
Xinjie Fan
Bo Chen
Mingyuan Zhou

title: Bayesian attention belief networks

authors:
Hao Zhang
Bo Chen
Yulai Cong
Dandan Guo
Hongwei Liu
Mingyuan Zhou

title: Deep autoencoding topic model with scalable hybrid Bayesian inference

venue: To appear in IEEE TPAMI,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

title: Stochastic backpropagation and variational inference in deep latent Gaussian models

title: Wordnet. In Theory and applications of ontology: computer applications, pages 231–243

authors:
Adji B Dieng
Francisco JR Ruiz
David M Blei

title: Topic modeling in embedding spaces

venue: Transactions of the Association for Computational Linguistics,

title: The Poisson gamma belief network

authors:
Matthew Hoffman
Francis R Bach
David M Blei

title: Online learning for latent Dirichlet Allocation

venue: In advances in neural information processing systems,

authors:
Guangxu Xun
Vishrawas Gopalakrishnan
Fenglong Ma
Yaliang Li
Jing Gao
Aidong Zhang

title: Topic discovery for short texts using word embeddings

venue: IEEE 16th international conference on data mining (ICDM),

authors:
Nikolaos Aletras
Mark Stevenson

title: Evaluating topic coherence using distributional semantics

venue: In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)– Long Papers,

authors:
Laurens Van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI blog,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Joseph Redmon
Ali Farhadi

title: Yolo9000: better, faster, stronger

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

heading: 1 Introduction

text:
Topic models, which have the ability to uncover the hidden semantic structure in a text corpus, have been widely applied to text analysis. Generally, a topic model is designed to discover a set of semantically-meaningful latent topics from a collection of documents, each of which captures word co-occurrence patterns commonly observed in a document. While vanilla topic models, such as latent Dirichlet allocation (LDA) [1] and Poisson factor analysis (PFA) [2], are able to achieve this goal, a series of their hierarchical extensions [3–12] have been developed in the hope of exploring multi-layer document representations and mining meaningful topic taxonomies. Commonly, these hierarchical topic models aim to learn a hierarchy, in which the latent topics exhibit an increasing level of abstraction when moving towards a deeper hidden layer, as shown in Fig. 1(a). Consequently, it provides users with an intuitive and interpretable way to better understand textual data.
Despite their attractive performance, many existing hierarchical topic models are purely data-driven and incorporate no prior domain knowledge, which may result in some learned topics failing to describe a semantically coherent concept [13], especially for those at a deeper hidden layer [14]. Furthermore, the inflexibility of adding prior knowledge also somewhat limits the applicability of hierarchical topic models, since it is common that a user is interested in a specific topic structure and only focused on information related to it [15]. To address this issue, we assume a structured prior in the form of a predefined topic hierarchy, where each topic is described by a semantic concept, with the concepts at adjacent layers following the hypernym relations, as illustrated in Fig. 1(b). Such a ∗Corresponding Author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
hierarchy can be easily constructed, either by some generic knowledge graph like WordNet [16] or according to the user’s customization. However, there are two main challenges: one is how to model this semantic hierarchy, and the other is how to combine it with topic models.
For the first challenge, a general and powerful approach [17] is to learn a distance-preserving mapping, which maps semantically similar topics to nearby points in the embedding space, with a symmetric distance measure (e.g., Euclidean or cosine distance) typically being used. However, the embeddings learned by such a scheme can not perfectly reflect the entailment between concepts. To mitigate this issue, Vendrov et al. [18] exploit the partial order structure of the semantic hierarchy to learn order embeddings that respect this partial order structure. On the other hand, considering that probability distributions are better at capturing uncertainties of concepts than point vectors, Athiwaratkun and Wilson [19] introduce density order embeddings as an improvement to order embeddings. Density order embeddings allows the entailment relationship to be expressed naturally, with general words such as “entity” corresponding to broad distributions that encompass more specific words such as “animal.” Through encapsulation of probability densities, it can intuitively reflect the hierarchical structure of semantic concepts, thus offering us an effective way to model the topic hierarchy.
As for the second challenge of combining the constructed topic hierarchy with topic models, there are two major difficulties. For one thing, the structured prior requires modeling the relationship between topics across different layers directly, while hierarchical topic models often assume that the topics at different layers are independently drawn in the prior [6], which makes the fusion of the two a bit far-fetched. For another, since the constructed topic hierarchy can be very deep for large-scale text corpora, an equally deep topic model with multiple latent layers is needed to incorporate with it. However, most existing deep neural topic models are hard to go very deep due to the problem of posterior collapse [14, 20–25]. Fortunately, SawETM [14], a recently proposed neural topic model, not only builds the dependency between topics at different layers in a shared embedding space, but also alleviates the issue of posterior collapse to some extent through a novel network module referred to as Sawtooth Connection (SC), and hence overcoming both difficulties.
In this paper, a novel knowledge-based hierarchical topic model has been proposed, the core idea of which is to represent each topic as a Gaussian embedding, and then project the topics of all layers into a shared embedding space to facilitate the injection of structured prior knowledge into the model. In particular, we first introduce the Gaussian-distributed embeddings to the SawETM, resulting in a variant of SawETM with a stochastic decoder called Gaussian SawETM. To incorporate prior belief into Gaussian SawETM, we then constrain the topics at adjacent layers to encourage them to capture the concepts satisfying the predefined hypernym relations. With auto-encoding variational inference, the entire model is learned in an end-to-end manner by minimizing the evidence lower bound and a regularization term. Extensive experiments demonstrate our model has competitive performance and better interpretability in comparison to most existing hierarchical topic models.
The main contributions of the paper can be summarized as follows:
• To capture semantic uncertainties of words and topics, we propose a novel probabilistic generative model with a stochastic decoder, referred to as Sawtooth Factorial Gaussian Topic Embeddings guided gamma belief network (Gaussian SawETM).
• To incorporate the structured prior knowledge from the real world, we propose TopicNet, a novel knowledge-based hierarchical topic model based on Gaussian SawETM.
• In addition to detailed theoretical analysis, we conduct extensive experiments to verify the effectiveness of the above two models. One of the most appealing properties of TopicNet is
its interpretability. We conduct extensive qualitative evaluation on the quality of the topics discovered by TopicNet, including word embedding, topic embedding, and topic hierarchies.

heading: 2 Gaussian SawETM

text: In this section, we elaborate on the construction and inference of Gaussian SawETM, a deep hierarchical topic model that represents words and topics as Gaussian-distributed embeddings [26].

heading: 2.1 Symmetric similarity: expected likelihood kernel

text:
The distance measure plays a key role in quantifying the similarities between embeddings. A simple choice is taking the inner product between the means of two Gaussians, which, however, does not take the advantage of the semantic uncertainties brought by Gaussian-distributed embeddings. Here we employ the expected likelihood kernel [19, 26, 27] as our similarity measure, which is defined as
E(s) (αi,αj) = ∫ x∈Rn N (x;µi,Σi)N ( x;µj ,Σj ) dx = N ( 0;µi − µj ,Σi+Σj ) , (1)
whereαi is a Gaussian distribution with mean µi and diagonal covariance matrix Σi. As a symmetric similarity function, this kernel considers the impact of semantic uncertainties brought by covariances.

heading: 2.2 Document decoder with sawtooth factorization and Gaussian embeddings

text:
The generative network (also known as decoder) is one of the core components of topic models. As discussed in Section 1, to build dependency between topics at two adjacent layers and learn a deep topic hierarchy, we draw experience from the Sawtooth Connection (SC) in SawETM. We develop a stochastic decoder by introducing Gaussian-distributed embeddings to better represent the topics in SawETM. Formally, the generative model with T latent layers can be expressed as
x (1) j ∼ Pois(Φ (1)θ (1) j ), { θ (t) j ∼ Gam(Φ (t+1)θ (t+1) j , 1/c (t+1) j ) }T−1 t=1 ,θ (T ) j ∼ Gam(γ, 1/c
(T+1) j ),{
Φ (t) k = Softmax
( log ( E(s)(α(t−1),α(t))k ))}T t=1 , { α (t) k ∼ N (x;µ (t) k ,Σ (t) k ) }T t=0 ,
(2)
In the above formula, xj ∈ ZV denotes the word count vector of the jth document, which is factorized as the product of the factor loading matrix Φ(1) ∈ RV×K1+ and gamma distributed factor score θ(1)j under the Poisson likelihood; and to obtain a multi-layer document representation, the hidden units θ(t)j ∈ R Kt + of the t th layer are further factorized into the product of the factor loading Φ(t+1) ∈ RKt×Kt+1+ and hidden units of the next layer. The top-layer hidden units θ (T ) j are sampled from a prior distribution. In addition, the kth topic α(t)k at layer t follows a Gaussian distribution with mean µ(t)k and covariance Σ (t) k , where the mean vector describes a semantic concept and the covariance matrix reflects its level of abstraction. Note that in the bottom layer α(0) represents the distributed embeddings of words. And Φ(t)k , used to capture the relationships between topics at two adjacent layers, is calculated based on the topic representations in the shared embedding space, instead of being sampled from a Dirichlet distribution. In particular, in the equation E(s)(·) refers to the symmetric similarity function defined in Eq. (1).

heading: 2.3 Document encoder: Weibull upward and downward encoder networks

text:
As presented in Section 2.2, instead of using Gaussian latent variables like most of neural topic models [28], our generative model employs the gamma distributed latent variables that are more suitable for modeling sparse and non-negative document representations. While in sampling based inference, the gamma distribution is commonly used to represent the conditional posterior of these latent variables, the difficulty of reparameterizing a gamma distributed random variable makes it difficult to apply it to an inference network [29, 30]. For this reason, we utilize a Weibull upwarddownward variational encoder inspired by the work in Zhang et al. [25, 31]. We let
q(θ (t) j | −) = Weibull(k (t) j ,λ (t) j ), (3)
where, the parameters k(t)j ,λ (t) j ∈ RKt are deterministically transformed from both the observed document features and the information from the stochastic up-down path θ(t+1)j . In detail, the inference network can be expressed as
h ′(0) j = ReLU(W (0) 1 xj + b (0) 1 )) h ′(t) j = h ′(t−1) j + ReLU(W (t) 1 h ′(t−1) j + b (t) 1 ), t = 1, · · · , T,
h (t) j = h ′(t) j ⊕Φ (t+1)θ (t+1) j , t = 0, · · · , T − 1, h (T ) j = h ′(T ) j ,
k (t) j = Softplus(W (t) 2 h (t) j + b (t) 2 ), λ (t) j = Softplus(W (t) 3 h (t) j + b (t) 3 ), t = 0, · · · , T,
(4)
where {b(t)i } 3,T i=1,t=1 ∈ RKt , {W (t) i } 3,T i=1,t=1 ∈ RKt×Kt−1 , and {h (t) j } N,T j=1,t=1 ∈ RKt , ⊕ denotes the concatenation in feature dimension, ReLU(·) = max(0, ·) is the nonlinear activation function, and Softplus(·) applies ln[1+ exp(·)] to each element to ensure positive shape and scale parameters of the Weibull distribution. To reduce the risk of posterior collapse, a skip-connected deterministic upward path [14] is used to obtain the hidden representations {h ′(t) j } N,T j=1,t=1 of the input xj .

heading: 2.4 Inference and estimation

text:
Similar to variational auto-encoders (VAEs) [32, 33], the training objective of Gaussian SawETM is to maximize the Evidence Lower Bound (ELBO):
LELBO = N∑ j=1 E [ ln p(xj |Φ(1),θ(1)j ) ] − N∑ j=1 T∑ t=1 E [ ln
q(θ (t) j |−)
p(θ (t) j |Φ(t+1),θ (t+1) j )
] , (5)
where q(θ(t)j |−) is the variational Weibull distribution in Eq. (3), and p(θ (l) j ) is the prior gamma distribution in Eq. (2). The first term is the expected log-likelihood or reconstruction error, while the second term is the Kullback–Leibler (KL) divergence that constrains q(θ(l)j ) to be close to its prior p(θ (l) j ) in the generative model. The analytic KL expression and simple reparameterization of the Weibull distribution make it simple to estimate the gradient of the ELBO, with respect to {µ(t)k }Tt=0 and {Σ(t)k }Tt=0 of Gaussian embeddings and other parameters in the inference network.

heading: 3 TopicNet: Semantic graph-guided topic discovery

text: In Section 2, a novel hierarchical topic model called Gaussian SawETM is proposed to discover meaningful topics organized into a hierarchy. Below we describe our solution to incorporate prior belief, e.g., knowledge graph, into Gaussian SawETM to guide the learning of the topic hierarchy.

heading: 3.1 TopicTree: Semantic graph constructed by prior knowledge

text:
Finding an appropriate way to encode prior knowledge is especially important for realizing the incorporation of prior belief into topic models. Due to the hierarchical structure of deep topic models,
correspondingly, we represent prior knowledge in the form of a hierarchical semantic graph and we call it TopicTree. Particularly, in the TopicTree each node is described by a semantic concept and the concepts of nodes at two adjacent layers follow the entailment relations, which results in a bottom-up process of gradual abstraction, as shown in Fig. 2(b). Besides, it is worth mentioning that some generic knowledge graphs such as WordNet [16] provide us a convenient way to build the TopicTree, since the hypernyms or hyponyms of any given word can be easily found in them.

heading: 3.2 Partial order: Asymmetric entailment relationship

text:
As described in Section 3.1, we organize prior knowledge into a hierarchical semantic graph so that it can be incorporated into our Gaussian SawETM naturally. Specifically, for a constructed TopicTree with depth T , we build a Gaussian SawETM with T latent layers corresponding to it, then we match the topics of each layer in the Gaussian SawETM with the semantic concepts of the corresponding layer in the TopicTree, i.e., each topic has a corresponding semantic concept to describe it. Most significantly, the relationships between semantic concepts in the TopicTree should also be injected to guide the learning of the topic hierarchy. Since the Gaussian SawETM projects the topics of all layers to a shared embedding space, such a goal can be achieved by imposing constraints on the topic representations in the shared embedding space.
Inspired by Vendrov et al. [18], the semantic hierarchy (akin to the hypernym relation between words) of the TopicTree can be seen as a special case of the partial order structure. In detail, a partial order over a set of points X is a binary relation such that for a, b, c ∈ X , the following properties hold: 1) a a (reflexively); 2) if a b and b a then a = b (anti-symmetry); 3) if a b and b c then a c (transitivity). Vendrov et al. [18] propose learning such asymmetric relationships with order embeddings: they represent semantic concepts as vectors of non-negative coordinates, and stipulate that smaller coordinates imply higher position in the partial order and correspond to a more general concept. For example, the following expression defines an ordered pair (x, y) of vectors in RN+ :
x y if and only if N∧ i=1 xi ≥ yi. (6)
Since the order embedding condition defined by Eq. (6) is too restrictive to satisfy by imposing a hard constraint, Vendrov et al. [18] relax the condition and turn to seek an approximate order-embedding. In particular, for an ordered pair of points, they apply a penalty that is formulated as:
E(x,y) = ‖max(0,y − x)‖2. (7) Note that this imposes a strong prior on the embeddings, which could encourage the learned relations to satisfy the partial order properties of transitivity and antisymmetry.
Although Gaussian-distributed embeddings are used to represent topics in our Gaussian SawETM, we find that they are also natural at capturing the partial order structure (or the semantic hierarchy of the TopicTree) [19]. For instance, for two topics following the hypernym relation, their Gaussiandistributed embeddings αi and αj form an ordered pair. Assuming that αi represents the specific topic and αj represents the general one, then we have αi αj , meanwhile we expect that αj corresponds to a broad distribution that encompasses αi. To meet this expectation, the KL divergence between αi and αj provides us a ready-made tool:
E (a) (αi,αj) = DKL (N i ‖N j) = ∫ x∈Rn N (x;µi,Σi) log (N (x;µi,Σi)/N (x;µj ,Σj)) dx
= 1
2
( tr ( Σ−1j Σi ) + (µi − µj)TΣ−1j (µi − µj)− d+ log (det (Σj)/det (Σi)) ) However, using E(a)(αi,αj) directly as a penalty for violating the partial order is undesirable. Since the KL divergence has a property that E(a)(αi,αj) = 0 if and only if αi = αj , which means the penalty is zero only if αi = αj , but we expect the penalty should also be zero when αi 6= αj and αi αj . For this reason, following Athiwaratkun and Wilson [19], we consider using a thresholded divergence as our penalty for order violation:
dγ (αi,αj) = max ( 0,E (a)(αi,αj)− γ ) . (8)
So this penalty can be zero if αi is properly encapsulated in αj . Nevertheless, note that we no longer have the strict partial order with such a penalty. In other words, the transitivity and anti-symmetry are not guaranteed. Therefore, actually our learned relations also respect an approximate order structure.

heading: 3.3 Objective: ELBO with a prior regularity

text:
Essentially, TopicNet is a topic model, so the ELBO still plays a leading role in the training objective. At the same time, a regularization term is also necessary for injecting hierarchical prior knowledge to guide the topic discovery. Therefore, the final objective of TopicNet can be written as:
LTopicNet = LELBO + βLprior, (9)
where β is a hyper-parameter used to balance the impact of these two terms. Particularly, for the ELBO term LELBO, it is the same as defined by Eq. (5). As for the regularization term Lprior, we cannot only constrain topics that follow the hypernym relation between two adjacent layers, but the topics that do not follow the hypernym relation should also be considered. This is consistent with the notion of positive and negative sample pairs in contrastive learning. Based on this idea, we use a max-margin loss to encourage the largest penalty between positive sample pairs to be below the smallest penalty between negative sample pairs, which is defined as:
L (t+1) j = max(0,m−max {dγ (i, j)}i∈D +min {dγ (i, j)}i/∈D). (10)
This represents the penalty for the jth topic at (t+ 1)th layer, and D is the set of hyponym topics of topic j. For i ∈ D, (i, j) forms a positive pair, for i /∈ D, (i, j) forms a negative pair. Parameter m is the margin that determines the degree to which the positive and negative pairs are separated. Considering the penalties for all topics in each layer, we obtain the prior regularization term.
Lprior = ∑T−1
t=0 ∑K(t+1) j=0 L (t+1) j . (11)
With the final objective, TopicNet learns the topics influenced by both the word co-occurrence information and prior semantic knowledge, thus discovering more interpretable topics. Note that, in our experiments, the hyper-parameters are set as m = 10.0 and β = 1.0.

heading: 3.4 Model Properties

text:
In this part, we recap several good properties of TopicNet: Uncertainty, Interpretability, and Flexibility.
Effectiveness of Gaussian-distributed embeddings: Mapping a topic to a distribution instead of a vector brings many advantages, including: on one hand, better capturing uncertainty about a representation, where the mean indicates the semantics of a topic and the covariance matrix reflects its level of abstraction; on the other hand, expressing asymmetric relationships more naturally than point vectors, which is beneficial to modeling the partial order structure of semantic hierarchy [19, 26].
Hierarchical interpretable topics: In conventional hierarchical topic models, the semantics of a topic is reflected by its distributed words, and we have to summarize a semantically coherent concept manually by post-hoc visualization. In some cases, the topic is too abstract to get a semantically coherent concept, which makes it hard to interpret. Such cases are common in deeper hidden layers. While in TopicNet, we know in advance what semantic concept a certain topic corresponds to, since it is encoded from prior knowledge and used to guide the topic discovery process of the model as a kind of supervision information. Therefore, once the model is trained, the words distributed under each topic are semantically related to the concept that describes it, as shown in Fig. 6. From this perspective, our TopicNet has better interpretability.
Flexibility to incorporate prior belief: Traditional topic models learn topics only by capturing word co-occurrence information from data, which do not consider the possible prior domain knowledge (e.g., Knowledge Graph like WordNet [34]). However, TopicNet provides a flexible framework to incorporate the structured prior knowledge into hierarchical topic models.

heading: 4 Experiments

text:
We conducted extensive experiments to verify the effectiveness of Gaussian SawETM and TopicNet. Our code is available at https://github.com/BoChenGroup/TopicNet.
Baseline methods and their settings: We compare Gaussian SawETM with the state-of-the-art topic models: 1. LDA Group, including: latent Dirichlet allocation(LDA) [1], which is a basic probabilistic topic model; LDA with Products of Experts (AVITM) [28], which replaces the mixture model in LDA with a product of experts and uses the AVI for training; Embedding Topic Model (ETM) [35], a generative model that incorporates word embeddings into traditional topic model. 2. DLDA Group, including: Deep LDA inferred by Gibbs sampling (DLDA-Gibbs) [36] and by TLASGR-MCMC (DLDA-TLASGR) [7]. 3. Deep Neural Topic Model (DNTM) Group, including Weibull Hybrid Autoencoding Inference model (WHAI) [25], which employs a deep variational encoder to infer hierarchical document representations, and Sawtooth Factorial Embedded Topic Model (SawETM) [14], where topics are modeled as learnable deterministic vectors. For all baselines, we use their official default parameters with best reported settings.
Datasets Our experiments are conducted on four widely-used benchmark datasets, including 20Newsgroups (20NG), Reuters Corpus Volume I (RCV1), Wikipedia (Wiki), and a subset of the Reuters-21578 dataset (R8), varying in scale and document length. 20NG, with a vocabulary of 2,000 words, has 20 classes and was split into 11,314 training and 7,532 test documents. RCV1 consists of 804,414 documents with a vocabulary size of 8,000. Wiki, with a vocabulary size of 10,000, consists of 3 million documents randomly downloaded from Wikipedia using the script provided by Hoffman et al. [37]. R8, with a vocabulary size of 10,000, has 8 classes and was split into 5,485 training and 2,189 test documents. The summary statistics of these datasets and other implementation details (such as dataset preprocessing and length statistics) are described in the Appendix.

heading: 4.1 Unsupervised learning for document representation

text:
Per-heldout-word perplexity: To measure predictive quality of the proposed model, we calculate the per-heldout-word perplexity (PPL) [7], on three regular document datasets, e.g., 20NG, RCV1, and Wiki. For deep topic models in the DLDA and DNTM groups, we report the PPL with 3 different stochastic layers T ∈ {1, 5, 15}. Specifically, for a 15-layer model, the topic size from bottom to top is set as K = [256, 224, 192, 160, 128, 112, 96, 80, 64, 56, 48, 40, 32, 16, 8], and the detailed description can be found in the Appendix. For each corpus, we randomly select 80% of the word tokens from each document to form a training matrix X, holding out the remaining 20% to form a testing matrix Y. We use X to train the model and calculate the per-held-word perplexity as
exp { − 1y.. V∑ v=1 N∑ n=1 yvn ln ∑S s=1 ∑K1 k=1 φ (1)s vk θ (1)s kn∑S s=1 ∑V v=1 ∑K1 k=1 φ (1)s vk θ (1)s kn } ,
where S is the total number of collected samples and y·· = ∑V v=1 ∑N n=1 yvn. As shown in Tab. 1, overall, DLDA-Gibbs perform best in terms of PPL, which is not surprising as they can sample from the true posteriors given a sufficiently large number of Gibbs sampling iterations. DLDA-TLASGR is a mini-batch based algorithm that is much more scalable in training than DLDA-Gibbs, at the expense of slightly degraded performance in out-of-sample prediction. Apart from these deep probabilistic topic models, Gaussian SawETM outperforms other neural topic models in all datasets. In particular, compared with Gaussian based single layer neural topic model, WHAI performs better, which can be attributed to two aspects, the first is the effectiveness of Weibull distribution in modeling sparse and non-negative document latent representations, and the second is aggregating the information in both the higher (prior) layer and that upward propagated to the current layer via inference network. Benefiting from the embedding decoder and the Sawtooth Connection that builds the dependency
of hierarchical topics, SawETM performs better than WHAI, especially at deeper layers. However, modeling topics with point vectors, SawETM cannot capture the uncertainties of topics. Gaussian SawETM represents the topics with Gaussian-distributed embeddings to capture the uncertainties, and achieves promising improvements over SawETM, especially for modeling complex and big corpora such as Wiki. Note that although the methods in DLDA group get better performance, they require iteratively sampling to infer latent document representations in the test stage, which limits their application [25]. By contrast, Gaussian SawETM can infer latent representations via direct projection, which makes it both scalable to large corpora and fast in out-of-sample prediction.
Document Classification & Clustering Document-topic distributions can be viewed as unsupervised document representations, to evaluate their quality, we perform document classification and clustering tasks. In detail, we first use the trained topic models to extract the latent representations of the testing documents and then use logistic regression to predict the label and use k-means to predict the clusters. we measure the classification performance by classification accuracy, and clustering performance by accuracy (ACC) and normalized mutual information (NMI), both of which are the higher the better. we test the model performance on 20NG and R8, where the document labels are considered. Different from the perplexity experiments, a larger vocabulary of 20,000 words is used for 20News to achieve better performance. The network structure is defined by TopicTree, which is introduced in Section 3.1, with the details deferred to the Appendix. The comparison results are summarized in Tab. 2. Benefiting from the use of Gaussian distributions on topics, Gaussian SawETM outperforms the other neural topic models on not only document classification but also clustering, demonstrating its effectiveness of extracting document latent representations. Note that Gaussian SawETM acquires promising improvements compared to SawETM in the 20NG dataset, showing the potential for modeling more complex data. With the predefined knowledge prior, TopicNet does not perform better, possibly because the introduced prior information does not well align with the end task. We leave the refinement of prior information for specific tasks to future study.

heading: 4.2 Interpretable topic discovery

text:
Apart from excelling at document representation learning, another appealing characteristic of TopicNet is that it can discover interpretable hierarchical topics. In this section, we perform the topic discovery experiment with a 7-layer TopicNet trained on 20NG. The network structure is constructed corresponding to the structure of TopicTree, which is set as [411, 316, 231, 130, 46, 11, 2].
Prior knowledge and corresponding learned topic: We show six example topics and their corresponding children nodes in Tab. 3, which illustrates how a particular concept is being represented in the 20 newsgroups. For example, the “military_action” concept, which has “assault”, “defense”, and “war” as its children nodes, has become tightly connected to the Waco Siege heavily discussed in this dataset; the “macromolecule” concept, which has “oil” as its single child node, has been represented by a topic related to vehicle, which is interesting as the 20newsgroups dataset has the
following two newsgropus— “rec.autos’ and “rec.motorcycles”—but not a newsgroup clearly related to biology and chemistry; and the “Atmosphere” concept, which has “Sky” as its single child node, has been represented by a topic related to NASA, reflecting the fact that “sci.space” is one of the 20 news groups.
Topic quality: Two metrics are considered here to evaluate the topic quality. The first metric is topic coherence, which is computed by taking the average Normalized Pointwise Mutual Information (NPMI) of the top 10 words of each topic [39]. It provides a quantitative measure of the semantic coherence of a topic. The second metric is topic diversity [35], which suggests the percentage of unique words in the top 25 words of all topics. Diversity close to 1 means more sundry topics. Following Dieng et al. [35], we define topic quality as the product of topic coherence and topic diversity. Note that topic quality is affected by the topic size, so it makes sense to compare different models on the same layer. As shown in Fig. 3, Gaussian SawETM performs better compared to SawETM, which can be attributed to the Gaussian-distributed embeddings that capture the semantic uncertainties of topics. Incorporating the hierarchical prior knowledge, TopicNet further achieves a higher score than Gaussian SawETM. However, it gets a lower score compared to DLDA-Gibbs in the first two layers, as the layer goes deeper, TopicNet performs better. This is easy to understand since information from data reduces slightly in the shallow layers but severely in the deep layers.
Visualisation of embedding space: First of all, we use a 3-layer Gaussian SawETM with 2-dim embedding trained on 20NG for the Gaussian-distribution embedding visualization. The top 4 words from the 16th topic at 1th layer are visualized in Fig. 4(a). As we can see, the words under the same topic are closer in the embedding space, which demonstrates the learned embeddings are semantically similar. Apart from the semantics of words and topics, the shadow range describes the uncertainties and reflects the abstraction levels. Secondly, we visualize the means of Gaussian-distributed embeddings learned by TopicNet with T-SNE [40]. As shown in Fig. 4.2, in the shared embedding space, the words with similar semantics are close to each other, meanwhile they all locate near the topic that has similar semantics to them, which means the TopicNet effectively captures the prior semantic information and discovers more interpretable topics.
Hierarchical structure of TopicNet: Distinct from many existing topic models that rely on posthoc visualization to understand topics, we can easily understand the semantic of the whole stochastic network via the predefined TopicTree. Specially, the hierarchical structure of TopicNet is shown in Fig. 6. The semantic of each topic can be represented by the concept in the TopicTree, which is more interpretable compared to traditional topic models. This result further confirms our motivation described in Fig. 1.

heading: 5 Conclusion

text: To equip topic models with the flexibility of incorporating prior knowledge, this paper makes a meaningful attempt and proposes a knowledge-based hierarchical topic model — TopicNet. This model first explores the use of Gaussian-distributed embeddings to represent topics, and then develops a regularization term that guides the model to discover topics by a predefined semantic graph. Compared with other neural topic models, TopicNet achieves competitive results on standard text analysis benchmark tasks, which illustrate a promising direction for the future development of text analysis models toward better interpretability and practicability.

heading: Acknowledgments

text: Bo Chen acknowledges the support of NSFC (U21B2006 and 61771361), Shaanxi Youth Innovation Team Project, the 111 Project (No. B18039) and the Program for Oversea Talent by Chinese Central Government.

title: TopicNet: Semantic Graph-Guided Topic Discovery