summaries: The authors propose a new artificial neural network architecture that is derived from a human visual model (Mély et al., 2018). The original (human vision) model can explain some of the human visual illusions, specifically contextual ones. While the adaptation from this human visual model to artificial neural networks was previously done by (Linsley et. al., 2018a), in this paper the authors extend (Linsley et. al., 2018a) to better capture some of the constraints in the human visual model, and also to add a formulation that can also model top-down connections (across layers).

summaries: The paper introduces a complex hierarchical recurrent model for contour detection loosely inspired by the organization of cortical circuits. Their model performs state-of-the-art on sample-limited versions of popular contour detection (BSDS500) and cell segmentation (SNEMI3D) datasets, and it reproduces the well-known tilt illusion when transfer-learning orientation estimation. Interestingly, "untraining" the tilt illusion degrades performance on contour detection.

summaries: The presented paper introduces a novel neural network architecture to explore the question whether visual illusions are corner cases of the human visual system, or whether they represent limitations of perception. The developed recurrent network architecture aims at being more sample efficient than existing methods. The findings discussed in the paper suggest that visual illusions are a byproduct of neural circuits that help to increase the robustness of the human visual system, which in turn suggests that neural networks for processing visual data could benefit from integrating circuits in similar ways. While existing work predominantly aims at explaining whether visual illusions are features or artifacts of the visual system, this work focuses on finding a computational solution to support the hypothesis that visual illusions are features. In particular, the contributions of this work are: (1) novel neural network architecture, called \gamma-networks, which is derived from the work of [Meley et al. 2018] and (2) that the proposed architecture is more sample efficient than SOTA convolutional architectures on contour detection tasks.

abstractText: We introduce a deep recurrent neural network architecture that approximates known visual cortical circuits (Mély et al., 2018). We show that this architecture, which we refer to as the γ-Net, learns to solve contour detection tasks with better sample efficiency than state-of-the-art feedforward networks, while also exhibiting a classic perceptual illusion, known as the orientation-tilt illusion. Correcting this illusion significantly reduces γ-Net contour detection accuracy by driving it to prefer lowlevel edges over high-level object boundary contours. Overall, our study suggests that the orientation-tilt illusion is a byproduct of neural circuits that help biological visual systems achieve robust and efficient contour detection, and that incorporating such circuits in artificial neural networks can improve computer vision.

authors:
P. Arbeláez
M. Maire
C. Fowlkes
J. Malik

title: Contour detection and hierarchical image segmentation

venue: IEEE Trans. Pattern Anal. Mach. Intell.,

authors:
K.L. Briggman
D.D. Bock

title: Volume electron microscopy for neuronal circuit reconstruction

venue: Curr. Opin. Neurobiol.,

authors:
S.N. Chettih
C.D. Harvey

title: Single-neuron perturbations reveal feature-specific competition

venue: in V1. Nature,

authors:
T. Cooijmans
N. Ballas
C. Laurent
Ç. Gülçehre
A. Courville

title: Recurrent batch normalization

venue: In International Conference on Learning Representations,

authors:
R. Deng
C. Shen
S. Liu
H. Wang
X. Liu

title: Learning to predict crisp boundaries

venue: In Computer Vision – ECCV

authors:
H. Ding
R.G. Smith
A. Poleg-Polsky
J.S. Diamond
K.L. Briggman

title: Species-specific wiring for direction selectivity in the mammalian retina

venue: Nature,

authors:
D.J. Felleman
D.C. Van Essen

title: Distributed hierarchical processing in the primate cerebral cortex

venue: Cereb. Cortex,

authors:
A.M. Fyall
Y. El-Shamayleh
H. Choi
E. Shea-Brown
A. Pasupathy

title: Dynamic representation of partially occluded objects in primate prefrontal and visual

venue: cortex. Elife,

authors:
R. Geirhos
C.R.M. Temme
J. Rauber
H.H. Schütt
M. Bethge
F.A. Wichmann

title: Generalisation in humans and deep neural networks

venue: Advances in Neural Information Processing Systems

authors:
D. George
W. Lehrach
K. Kansky
M. Lázaro-Gredilla
C. Laan
B. Marthi
X. Lou
Z. Meng
Y. Liu
H. Wang
A. Lavin
D.S. Phoenix

title: A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs

authors:
C.D. Gilbert
M. Sigman

title: Brain states: top-down influences

venue: in sensory processing. Neuron,

authors:
R.H. Hahnloser
R. Sarpeshkar
M. a. Mahowald
R.J. Douglas
H.S. Seung

title: Digital selection and analogue amplification coexist in a cortex-inspired silicon

venue: circuit. Nature,

authors:
S. Hallman
C.C. Fowlkes

title: Oriented edge forests for boundary detection

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
J. He
S. Zhang
M. Yang
Y. Shan
T. Huang

title: Bi-Directional cascade network for perceptual edge detection

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
K. He
X. Zhang
S. Ren
J. Sun

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
J.C. Heck
F.M. Salem

title: Simplified minimal gated unit variations for recurrent neural networks

venue: IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS),

authors:
S. Hochstein
M. Ahissar

title: View from the top: Hierarchies and reverse hierarchies in the visual system

authors:
M. Januszewski
V. Jain

title: Segmentation-Enhanced CycleGAN. February 2019

authors:
H. Kafaligonul
B.G. Breitmeyer
H. Öğmen

title: Feedforward and feedback processes in vision

venue: Front. Psychol.,

authors:
K. Kar
J. Kubilius
K. Schmidt
E.B. Issa
J.J. DiCarlo

title: Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition

authors:
N. Kasthuri
K.J. Hayworth
D.R. Berger
R.L. Schalek
J.A. Conchello
S. Knowles-Barley
D. Lee
A. Vázquez-Reina
V. Kaynig
T.R. Jones
M. Roberts
J.L. Morgan
J.C. Tapia
H.S. Seung
W.G. Roncal
J.T. Vogelstein
R. Burns
D.L. Sussman
C.E. Priebe
H. Pfister
J.W. Lichtman

title: Saturated reconstruction of a volume of neocortex

authors:
S.W. Keemink
M.C.W. van Rossum

title: A unified account of tilt illusions, association fields, and contour detection based on elastica

venue: Vision Res.,

authors:
T.C. Kietzmann
C.J. Spoerer
L. Sörensen
others

title: Recurrence required to capture the dynamic computations of the human ventral visual stream

venue: arXiv preprint arXiv,

authors:
J.K. Kim
M. Ricci
T. Serre

title: Not-So-CLEVR: learning same–different relations strains feedforward neural networks. Interface Focus theme issue on “Understanding images in biological and computer vision

authors:
D.P. Kingma
J. Ba

venue: arXiv preprint arXiv:1412.6980,

authors:
P.C. Klink
B. Dagnino
M.-A. Gariel-Mathis
P.R. Roelfsema

title: Distinct feedforward and feedback effects of microstimulation in visual cortex reveal neural mechanisms of texture segregation

title: Pushing the boundaries of boundary detection using deep learning

venue: arXiv preprint arXiv:1511.07386,

title: Deep neural networks: A new framework for modeling biological vision and brain information processing

venue: Annu Rev Vis Sci,

authors:
B.M. Lake
R. Salakhutdinov
J.B. Tenenbaum

title: Human-level concept learning through probabilistic program induction

venue: Science,

authors:
K. Lee
J. Zung
P. Li
V. Jain
H. Sebastian Seung

title: Superhuman accuracy on the SNEMI3D connectomics challenge

venue: In Neural Information Processing Systems,

authors:
T.S. Lee
D. Mumford

title: Hierarchical bayesian inference in the visual cortex

venue: J. Opt. Soc. Am. A Opt. Image Sci. Vis.,

authors:
Q. Liao
T. Poggio

title: Bridging the gaps between residual learning, recurrent neural networks and visual cortex

authors:
D. Linsley
J.K. Kim
V. Veerabadran
C. Windolf
T. Serre

title: Learning long-range spatial dependencies with horizontal gated recurrent units

venue: In Neural Information Processing Systems (NIPS),

authors:
D. Linsley
J. Kim
D. Berson
T. Serre

title: Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks. November 2018b

authors:
D. Linsley
J. Kim
V. Veerabadran
C. Windolf
T. Serre

authors:
D. Linsley
D. Shiebler
S. Eberhardt
T. Serre

title: Learning what and where to attend with humans in the loop

authors:
Y. Liu
M.-M. Cheng
X. Hu
J.-W. Bian
L. Zhang
X. Bai
J. Tang

title: Richer convolutional features for edge detection

authors:
W. Lotter
G. Kreiman
D. Cox

title: Deep predictive coding networks for video prediction and unsupervised learning

authors:
D.A. Mély
D. Linsley
T. Serre

title: Complementary surrounds explain diverse contextual phenomena across visual modalities

venue: Psychol. Rev.,

title: Induction of multiscale temporal structure

authors:
A. Nayebi
D. Bear
J. Kubilius
K. Kar
S. Ganguli
D. Sussillo
J.J. DiCarlo
D.L. Yamins

title: Task-Driven convolutional recurrent models of the visual system

authors:
J. Nunez-Iglesias
R. Kennedy
T. Parag
J. Shi
D.B. Chklovskii

title: Machine learning of hierarchical clustering to segment 2D and 3D images

venue: PLoS One,

authors:
R.C. O’Reilly
D. Wyatte
S. Herd
B. Mingus
D.J. Jilk

title: Recurrent processing during object recognition

authors:
B. O’Toole
P. Wenderoth

title: The tilt illusion: repulsion and attraction effects in the oblique meridian

authors:
P.J. Phillips
A.N. Yates
Y. Hu
C.A. Hahn
E. Noyes
K. Jackson
J.G. Cavazos
G. Jeckeln
R. Ranjan
S. Sankaranarayanan
J.-C. Chen
C.D. Castillo
R. Chellappa
D. White
A.J. O’Toole

title: Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms

venue: Proc. Natl. Acad. Sci. U. S. A.,

authors:
B. Recht
R. Roelofs
L. Schmidt
V. Shankar

title: Do CIFAR-10 classifiers generalize to CIFAR-10

title: Cortical algorithms for perceptual grouping

venue: Annu. Rev. Neurosci.,

authors:
O. Ronneberger
P. Fischer
T. Brox

title: U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention

venue: MICCAI

authors:
A. Rosenfeld
R. Zemel
J.K. Tsotsos

title: The elephant in the room

title: Deep learning: The good, the bad, and the ugly

authors:
M. Siegel
T.J. Buschman
E.K. Miller

title: Cortical information flow during flexible sensorimotor decisions

authors:
C.J. Spoerer
P. McClure
N. Kriegeskorte

title: Recurrent convolutional neural networks: A better model of biological object recognition

authors:
C. Tallec
Y. Ollivier

title: Can recurrent neural networks warp time

authors:
H. Tang
M. Schrimpf
W. Lotter
C. Moerman
A. Paredes
J. Ortega Caro
W. Hardesty
D. Cox
G. Kreiman

title: Recurrent computations for visual pattern completion

authors:
D. Ulyanov
A. Vedaldi
V. Lempitsky

title: Instance normalization: The missing ingredient for fast stylization

authors:
E. Vorontsov
C. Trabelsi
S. Kadoury
C. Pal

title: On orthogonality and learning recurrent networks with long term dependencies

authors:
Y. Wang
X. Zhao
Y. Li
K. Huang

title: Deep crisp boundaries: From boundaries to Higher-Level tasks

venue: IEEE Trans. Image Process.,

authors:
H. Wen
K. Han
J. Shi
Y. Zhang
E. Culurciello
Z. Liu

title: Deep predictive coding network for object recognition

authors:
D. Wyatte
D.J. Jilk
R.C. O’Reilly

title: Early recurrent feedback facilitates visual object recognition under challenging conditions

authors:
S. Xie
Z. Tu

title: Holistically-Nested edge detection

venue: Int. J. Comput. Vis.,

authors:
D.L.K. Yamins
J.J. DiCarlo

title: Using goal-driven deep learning models to understand sensory cortex

venue: Nat. Neurosci.,

title: MS∗, before elementwise multiplication with A . All results in the main text use this implementation. Following the lead of (Linsley et al., 2018a), we incorporated normalizations into the fGRU. Let r ∈ R denote the vector of layer activations that will be normalized. We chose instance normalization (Ulyanov et al., 2016) since it is independent of batch size, which was 1 for γ-Nets

title: 2017, mean affinity agglomeration). B MEMBRANE PREDICTION MODELS Our reference model for membrane prediction is the 3D U-Net of (Lee et al., 2017). This architecture consists of four encoder blocks (multiple convolutions and skip connections, pooling

venue: 1c, Conv,−

venue: As in Lee et al

heading: 1 INTRODUCTION

text:
An open debate since the inception of vision science concerns why we experience visual illusions. Consider the class of “contextual” illusions, where the perceived qualities of an image region, such as its orientation or color, are biased by the qualities of surrounding image regions. A well-studied contextual illusion is the orientation-tilt illusion depicted in Fig. 1a, where perception of the central grating’s orientation is influenced by the orientation of the surrounding grating (O’Toole & Wenderoth, 1977). When the two orientations are similar, the central grating appears tilted slightly away from the surround (Fig. 1a, top). When the two orientations are dissimilar, the central grating appears tilted slightly towards the surround (Fig. 1a, bottom). Is the contextual bias of the orientation-tilt illusion a bug of biology or a byproduct of optimized neural computations?
Over the past 50 years, there has been a number of neural circuit mechanisms proposed to explain individual contextual illusions (reviewed in Mély et al., 2018). Recently, Mély et al. (2018) proposed a cortical circuit, constrained by physiology of primate visual cortex (V1), that offers a unified explanation for contextual illusions across visual domains – from the orientation-tilt illusion to color induction. These illusions arise in the circuit from recurrent interactions between neural populations with receptive fields that tile visual space, leading to contextual (center/surround) effects. For the orientation-tilt illusion, neural populations encoding the surrounding grating can either suppress or facilitate the activity of neural populations encoding the central grating, leading to repulsion vs. attraction, respectively. These surround neural populations compete to influence encodings of the central grating: suppression predominates when center/surround are similar, and facilitation predominates when center/surround are dissimilar.
The neural circuit of Mély et al. (2018) explains how contextual illusions might emerge, but it does not explain why. One possibility is that contextual illusions like the orientation-tilt illusion are “bugs”: vestiges of evolution or biological constraints on the neural hardware. Another possibility is that contextual illusions are the by-product of efficient neural routines for scene segmentation (Keemink & van Rossum, 2016; Mély et al., 2018). Here, we provide computational evidence for the latter
†These authors contributed equally to this work. ‡Currently at DeepMind, London, UK.
possibility and demonstrate that the orientation-tilt illusion reflects neural strategies optimized for object contour detection.
Contributions We introduce the γ-Net, a trainable and hierarchical extension of the neural circuit of Mély et al. (2018), which explains contextual illusions. (i) The γ-Net is more sample efficient than state-of-the-art convolutional architectures on two separate contour detection tasks. (ii) Similar to humans but not state-of-the-art contour detection models, the γ-Net exhibits an orientation-tilt illusion after being optimized for contour detection. This illusion emerges from its preference for high-level object-boundary contours over low-level edges, indicating that neural circuits involved in contextual illusions also support sample-efficient solutions to contour detection tasks.

heading: 2 RELATED WORK

text:
Modeling the visual system Convolutional neural networks (CNNs) are often considered the de facto “standard model” of vision. CNNs and their extensions represent the state of the art for most computer vision applications with performance approaching – and sometimes exceeding – human observers on certain visual recognition tasks (He et al., 2016; Lee et al., 2017; Phillips et al., 2018). CNNs also provide the best fit to rapid neural responses in the visual cortex (see Kriegeskorte 2015; Yamins & DiCarlo 2016 for reviews). Nevertheless, multiple lines of evidence suggest that biological vision is still far more robust and versatile than CNNs (see Serre, 2019, for a recent review). CNNs suffer from occlusions and clutter (Fyall et al., 2017; Rosenfeld et al., 2018; Tang et al., 2018). They are also sample inefficient at learning visual relations (Kim et al., 2018) and solving simple grouping tasks (Linsley et al., 2018c). State-of-the-art CNNs require massive datasets to reach their impressive accuracy (Lake et al., 2015) and their ability to generalize beyond training data is limited (Geirhos et al., 2018; Recht et al., 2018).
Cortical feedback contributes to the robustness of biological vision (Hochstein & Ahissar, 2002; Wyatte et al., 2014; Kafaligonul et al., 2015). Feedforward projections in the visual system are almost always matched by feedback projections (Felleman & Van Essen, 1991), and feedback has been implicated in visual “routines” that cannot be implemented through purely feedforward vision, such as incremental grouping or filling-in (O’Reilly et al., 2013; Roelfsema, 2006). There is a also a growing body of work demonstrating the potential of recurrent neural networks (RNNs) to account for neural recordings (Fyall et al., 2017; Klink et al., 2017; Siegel et al., 2015; Tang et al., 2018; Nayebi et al., 2018; Kar et al., 2019; Kietzmann et al., 2019).
Feedback for computer vision In contrast to CNNs, which build processing depth through a cascade of filtering and pooling stages with unique weights, RNNs process stimuli with filtering stages that reuse weights over “timesteps” of recurrence. On each discrete processing timestep, an RNN updates its hidden state through a nonlinear combination of an input and its the hidden state from its previous timestep. RNNs have been extended from their roots in sequence processing (e.g., Mozer 1992) to computer vision by computing the activity of RNN units through convolutional kernels. The common interpretation of these convolutional-RNNs, is that the input to each layer functions as a (fixed) feedforward drive, which is combined with layer-specific feedback from an evolving hidden state to dynamically adjust layer activity (Linsley et al., 2018c; George et al., 2017; Lotter et al., 2016; Wen et al., 2018; Liao & Poggio, 2016; Spoerer et al., 2017; Nayebi et al., 2018; Tang et al., 2018). In the current work, we are motivated by a similar convolutional-RNN, the horizontal gated recurrent unit (hGRU, Linsley et al. 2018a), which approximates the recurrent neural circuit model of (Mély et al., 2018) for explaining contextual illusions.

heading: 3 RECURRENT NEURAL MODELS

text:
We begin by reviewing the dynamical neural circuit of Mély et al. (2018). This model explains contextual illusions by simulating interactions between cortical hypercolumns tiling the visual field (where hypercolumns describe a set of neurons encoding features for multiple visual domains at a single retinotopic position). In the model, hypercolumns are indexed by their 2D coordinate (x, y) and feature channels k. Units in hypercolumns encode idealized responses for a visual domain (e.g., neural responses from the orientation domain were used to simulate an orientation-tilt illusion;
(a)
xyk) interactions between units in
a layer `, which receives feedforward drive from a center pathway encoding feature k (e.g., edges oriented at 0◦ or 22.5◦) at position (x, y) in an image. Blocks depict different layers, and arrowed connections denote top-down feedback. (c) A deep network schematic of the circuit diagram in (b), which forms the basis of the γ-Net introduced here. Horizontal and top-down connections are implemented with feedback gated recurrent units (fGRUs). Image encodings pass through these blocks on every timestep, from bottom-up (left path) to top-down (right path), and predictions are read out from the fGRU closest to image resolution on the final timestep. This motif can be stacked to create a hierarchical model.
Fig. 1b). Dynamics of a single unit at xyk obey the following equations (we bold activity tensors to distinguish them from learned kernels and parameters):
ηḢ (S) xyk + 2H (S) xyk = [ ξZxyk − (αH(F )xyk + µ)C (S) xyk ] + # Stage 1: Recurrent suppression of Z
τḢ (F ) xyk + σ 2H (F ) xyk =
[ νC
(F ) xyk ] + , # Stage 2: Recurrent facilitation of H(S)
where
C (S) xyk = (W S ∗H(F ))xyk # Compute suppression interactions
C (F ) xyk = (W F ∗H(S))xyk. # Compute facilitation interactions
Circuit activities consist of a feedforward drive, recurrent suppression, and recurrent facilitation, respectively denoted as Z,H(S),H(F ) ∈ RX×Y×K (X is width, Y is height of the tensor, and K is its feature channels)∗. The circuit takes its “feedforward” input Z from hypercolumns (e.g., orientation encodings from hypercolumn units), and introduces recurrent suppressive and facilitatory interactions between units, C(S),C(F ) ∈ RX×Y×K (Fig. 1b). These interactions are implemented with separate kernels for suppression and facilitation, WS ,WF ∈ RE×E×K×K, where E is the spatial extent of connections on a single timestep (connectivity in this model is constrained by primate physiology). ∗Suppression refers to interactions that reduce unit activity, and facilitation refers to interactions that increase activity. These computations can cause illusory repulsion or attraction at the level of neural populations.
These interactions are implemented through convolutions, allowing them to serially spread over timesteps of processing to connect units positioned at different spatial locations. The circuit outputs H(F ) after reaching steady state.
The circuit model of Mély et al. (2018) depends on competition between H(S) and H(F ) to explain the orientation-tilt illusion. Competition is implemented by (i) computing suppression vs. facilitation in separate stages, and (ii) having non-negative activities, which enforces these functionally distinct processing stages. With these constraints in the circuit model, the strength of recurrent suppression – but not facilitation – multiplicatively increases with the net recurrent output. For the orientation-tilt illusion, suppression predominates when center and surround gratings have similar orientations. This causes encodings of the surround grating to “repulse” encodings of the center grating. On the other hand, facilitation predominates (causing “attraction”) when center and surround gratings have dissimilar orientations because it is additive and not directly scaled by the circuit output.
Parameters controlling the circuit’s integration, suppression/facilitation, and patterns of horizontal connections between units are tuned by hand. Linear and multiplicative suppression (i.e., shunting inhibition) are controlled by scalars µ and α, feedforward drive is modulated by the scalar ξ, and linear facilitation is controlled by the scalar ν. Circuit time constants are scalars denoted by η, , τ and σ. All activities are non-negative and both stages are linearly rectified (ReLU) [·]+ = max(·, 0).
Feedback gated recurrent units Linsley et al. (2018a) developed a version of this circuit for computer vision applications, called the hGRU. In their formulation they use gradient descent (rather than hand-tuning like in the original circuit) to fit its connectivity and parameters to image datasets. The hGRU was designed to learn a difficult synthetic incremental grouping task, and a single layer of the hGRU learned long-range spatial dependencies that CNNs with orders-of-magnitude more weights could not. The hGRU replaced the circuit’s time constants with dynamic gates, converted the recurrent state H(S) for suppression into an instantaneous activity, and introduced a term for quadratic facilitation. The hGRU also relaxed biological constraints from the original circuit, including an assumption of non-negativity, which enforced competition between recurrent suppression vs. facilitation (e.g., guaranteeing that Stage 1 in the circuit model describes suppression of Zxyk).
We extend the hGRU formulation in two important ways. First, like Mély et al. (2018), we introduce non-negativity. This constraint was critical for Mély et al. (2018) to explain contextual illusions, and as we describe below, was also important for our model. Second, we extend the circuit into a hierarchical model which can learn complex contour detection tasks. Recent neurophysiological work indicates that contextual effects emerge from both horizontal and top-down feedback (Chettih & Harvey, 2019). Motivated by this, we develop versions of the circuit to simulate horizontal connections between units within a layer, and top-down connections between units in different layers.
We call our module the feedback gated recurrent unit (fGRU). We describe the evolution of fGRU recurrent units in H ∈ RX×Y×K, which are influenced by non-negative feedforward encodings Z ∈ RX×Y×K (e.g., a convolutional layer’s response to a stimulus) over discrete timesteps ·[t]:
Stage 1:
GS = sigmoid(US ∗H[t− 1]) # Compute channel-wise selection CS = WS ∗ (H[t− 1] GS) # Compute suppression interactions
S = [ Z− [ (αH[t− 1] + µ) CS ] + ] + , # Suppression of Z
Stage 2:
GF = sigmoid(UF ∗ S) # Compute channel-wise recurrent updates CF = WF ∗ S # Compute facilitation interactions
H̃ = [ ν(CF + S) + ω(CF ∗ S) ] +
# Facilitation of S
H[t] = (1−GF ) H[t− 1] + GF H̃. # Update recurrent state
Like the original circuit, the fGRU has separate stages for suppression (S) and facilitation (H). In the first stage, the feedforward encodings Z are suppressed by non-negative interactions between units in H[t− 1] (an fGRU hidden state from the previous timestep). Suppressive interactions are computed with the kernel WS ∈ RE×E×K×K, where E describes the spatial extent of horizontal connections on a single timestep. This kernel is convolved with a gated version of the persistent hidden state H[t−1]. The gate activity GS is computed by applying a sigmoid nonlinearity to a convolution of the kernel US ∈ R1×1×K×K with H[t− 1], which transforms its activity into the range [0, 1]. Additive and multiplicative forms of suppression are controlled by the parameters µ, α ∈ RK, respectively. In the second stage, additive and multiplicative facilitation is applied to the instantaneous activity S. The kernels WF ∈ RE×E×K×K controls facilitation interactions. Additive and multiplicative forms of facilitation are scaled by the parameters ν, ω ∈ RK, respectively. A gate activity is also computed during this stage to update the persistent recurrent activity H. The gate activity GF is computed by applying a sigmoid to a convolution of the kernel UF ∈ R1×1×K×K with S. This gate updates H[t] by interpolating H[t− 1] with the candidate activity H̃. After every timestep of processing, H[t] is taken as the fGRU output activity. As detailed in the following section, the fGRU output hidden state is either passed to the next convolutional layer (Fig. 1c, fGRU(`)−→ conv(`+1)), or used to compute top-down connections (Fig. 1c, fGRU(`+1)−→ fGRU(`)). The fGRU has different configurations for learning horizontal connections between units within a layer or top-down connections between layers (Fig. 1b). These two configurations stem from changing the activities used for a fGRU’s feedforward encodings and recurrent hidden state. “Horizontal connections” between units within a layer are learned by setting the feedforward encodings Z to the activity of a preceding convolutional layer, and setting the hidden state H to a persistent activity initialized as zeros (Fig. 1c, conv(`) −→ fGRU(`)). “Top-down connections” between layers are learned by setting fGRU feedforward encodings Z to the persistent hidden state H(`) of a fGRU at layer ` in a hierarchical model, and the hidden state H to the persistent activity H(`+1) of an fGRU at a layer one level up in the model hierarchy (Fig. 1c, fGRU(`+1) −→ fGRU(`)). The functional interpretation of the top-down fGRU is that it first suppresses activity in the lower layer using the higher layer’s recurrent horizontal activities, and then applies a kernel to the residue for facilitation, which allows for computations like interpolation, sharpening, or “filling in”. Note that an fGRU for top-down connections does not have a persistent state (it mixes high and low-level persistent states), but an fGRU for horizontal connections does.
γ-Net Our main objective is to test how a model with the capacity for contextual illusions performs on natural image analysis. We do this by incorporating fGRUs into leading feedforward architectures for contour detection tasks, augmenting their feedforward processing with modules for learning feedback from horizontal and top-down connections (Fig. 1c). We refer to the resulting hierarchical models as γ-Nets, because information flows in a loop that resembles a γ: image encodings make a full bottom-up to top-down cycle through the architecture on every timestep, until dense predictions are read-out from the lowest-level recurrent layer of the network (thus, information flows in at the top of the hierarchy, loops through the network, and flows out from the top of the hierarchy). In our experiments we convert leading architectures for two contour detection problems into γ-Nets: A VGG16 for BSDS500 (He et al., 2019), and a U-Net for detection of cell membranes in serial electron microscopy images (Lee et al., 2017). See Appendix A for an algorithmic description of γ-net.

heading: 4 CONTOUR DETECTION EXPERIMENTS

text:
Overview We evaluated γ-Net performance on two contour detection tasks: object contour detection in natural images (BSDS500 dataset; Arbeláez et al., 2011) and cell membrane detection in serial electron microscopy (SEM) images of mouse cortex (Kasthuri et al., 2015) and mouse retina (Ding et al., 2016). Different γ-Net configurations were used on each task, with each building on the leading architecture for their respective datasets. All γ-Nets use 8-timesteps of recurrence and instance normalization (normalization controls vanishing gradients in RNN training; Ulyanov et al., 2016; Cooijmans et al., 2017, see Appendix A for details). The γ-Nets were trained with Tensorflow and NVIDIA Titan RTX GPUs using single-image batches and the Adam optimizer (Kingma & Ba, 2014, dataset-specific learning rates are detailed below). Models were trained with early stopping, which
terminated training if the validation loss did not drop for 50 straight epochs. The weights with the best validation-set performance were used for testing.
Model evaluation We evaluated models in two ways. First, we validated them against state-of-theart models for each contour dataset using standard benchmarks. As discussed below, we verified that our implementations of these state-of-the-art models matched published performance. Second, we tested sample-efficiency after training on subsets of the contour datasets without augmentations. Sample-efficiency compares the inductive biases of different architectures, and is critical for understanding how the capacity for exhibiting contextual illusions influences performance. We report model “wall time” in Appendix A; however, the focus of our work is on sample efficiency rather than the hardware/software-level optimizations that influence wall time. Model performance is evaluated as the F-measure at the Optimal Dataset Scale across images after non-maximum suppression post-processing (F1-ODS; Arbeláez et al., 2011), as is standard for contour detection tasks.

heading: 4.1 OBJECT CONTOUR DETECTION IN NATURAL IMAGES

text:
Dataset We trained models for object contour detection on the BSDS500 dataset (Arbeláez et al., 2011). The dataset contains object-contour annotations for 500 natural images, which are split into train (200), validation (100), and test (200) sets.
Architecture details The leading approach to BSDS500 is the Bi-Directional Cascade Network (BDCN, He et al. 2019), which places multi-layer readout modules at every processing block in a
ILSVRC12-pretrained VGG16, and optimizes a loss that balances contributions from each of these readouts to achieve better scale tolerance in final prediction. All leading deep learning approaches to BSDS500 begin with a VGG16 pretrained on ILSVRC12 object recognition (He et al., 2019).
Our γ-Net for BSDS500 begins with the same ILSVRC12-pretrained VGG16 as the BDCN. fGRUs were introduced for learning horizontal (conv2_2, conv3_3, conv4_3, conv5_3) and top-down connections (conv5_3−→ conv4_3, conv4_3−→ conv3_3, and conv3_3−→ conv2_2). To pass topdown activities between layers, higher-level activities were resized to match lower-level ones, then passed through two layers of 1× 1 convolutions with linear rectification, which registered feature representations from higher-to-lower layers. The γ-Net was trained with learning rates of 3e−4 on its randomly initialized fGRU weights and 1e−5 on its VGG-initialized weights. Training time and parameter counts for this γ-Net and the BDCN is in Appendix Table S1.
In contrast to the BDCN (and other recent approaches to BSDS) with multiple read-outs and engineered loss functions, we take γ-Net predictions as a linear transformation of the lowest fGRU layer in its feature hierarchy, and optimize the model with binary cross entropy between per-pixel predictions and labels (following the method of Xie & Tu, 2017). This approach works because the γ-Net uses feedback to merge high- and low-level image feature representations at the bottom of its feature hierarchy, resembling classic “V1 scratchpad” hypotheses for computation in visual cortex (Gilbert & Sigman, 2007; Lee & Mumford, 2003). We compared γ-Nets with a BDCN implementation released by the authors, which was trained using the routine described in He et al. (2019)†.
†We replicated published results by following this routine and training the BDCN with the same regularization, batch size, and optimizer as He et al. (2019). For sample efficiency experiments, we searched through multiple learning rates and found that it had no affect on performance on these small BSDS500 datasets (Fig. S3).
Results We validated the γ-Net against the BDCN after training on a full and augmented BSDS training set (Xie & Tu, 2017). The γ-Net performed similarly in F1 ODS (0.802) as the BDCN (0.806) and humans (0.803), and outperformed all other approaches to BSDS (Fig. 2a; Deng et al. 2018; Xie & Tu 2017; Hallman & Fowlkes 2015; Kokkinos 2015; Wang et al. 2019; Liu et al. 2019).
Our hypothesis is that contextual illusions reflect routines for efficient scene analysis, and that the capacity for exhibiting such illusions improves model sample efficiency. Consistent with this hypothesis, the γ-Net was up to an order-of-magnitude more efficient than the BDCN. A γ-Net trained on 5% of BSDS performs on par with a BDCN trained on 10% of the BSDS, and a γ-Net trained on 10% of the BSDS performs on par with a BDCN trained on 100% of BSDS. Unlike the BDCN, the γ-Net trained on 100% of BSDS outperformed the state of the art for non-deep learning based models (Hallman & Fowlkes, 2015), and nearly matched the performance of the popular HED trained with augmentations (Xie & Tu, 2017). We also evaluated lesioned versions of γ-Net to measure the importance of its horizontal/top-down connections, recurrence, non-negativity, and different specifications of its fGRU recurrent modules for detecting contours in BSDS500 (Fig. S4).
We examined recurrent feedback strategies learned by γ-Net for object contour by visualizing its performance on every timestep of processing. This was done by passing its activity at a timestep through the final linear readout layer. The γ-Net iteratively refines its initially coarse contour predictions. For example, the top row of Fig. 2c shows that the γ-Net selectively enhances the boundaries around the runner’s bodies while suppressing the feature activities created by the crowd. In the next row of predictions, salient zebra stripes are gradually suppressed in favor of body contours (see Fig. S5 for γ-Net prediction dynamics and its tendency towards steady state solutions).

heading: 4.2 CELL MEMBRANE DETECTION

text:
Datasets “Connectomics” involves extracting the wiring diagrams of neurons from serial electron microscope (SEM) imaging data, and is an important step towards understanding the algorithms of brains (Briggman & Bock, 2012). CNNs can automate this procedure by segmenting neuron membranes in SEM images. Large-scale challenges like SNEMI3D (Kasthuri et al., 2015), which contains annotated images of mouse cortex, have helped drive progress towards automation. Here, we test models on membrane detection in SNEMI3D and a separate SEM dataset of mouse retina (“Ding” from Ding et al. 2016). We split both datasets into training (80 images for SNEMI3D and 307 images for Ding) and test sets (20 images for SNEMI3D and 77 images for Ding). Next, we generated versions of each training dataset with 100%, 10%, or 5% of the images, as well as versions of the full datasets augmented with random left-right and up-down flips (A+100%).
Architecture details The state-of-the-art on SNEMI3D is a U-Net variant (Ronneberger et al., 2015), which uses a different depth, different number of feature maps at every layer, and introduces new features like residual connections (Lee et al., 2017). We developed a γ-Net for cell membrane segmentation, which resembled this U-Net variant Lee et al. (2017) (see Appendix B for details). These γ-Net were trained from a random initialization with a learning rate of 1e−2 to minimize class-balanced per-pixel binary cross-entropy, and compared to the U-Net from Lee et al. (2017). Training time and parameter counts for these models is in Appendix Table S2.
Results The γ-Net and U-Net of Lee et al. (2017) performed similarly when trained on full augmented versions of both the SNEMI3D and Ding datasets (Fig. 3a A+100%). However, γ-Nets were consistently more sample efficient than U-Nets on every reduced dataset condition (Fig. 3b).
We visualized the recurrent membrane detection strategies of γ-Nets trained on 100% of both datasets. Membrane predictions were obtained by passing neural activity at every timestep through the final linear readout. The γ-Net prediction timecourse indicates that it learns a complex visual strategy for membrane detection: it gathers a coarse “gist” of membranes in the first timestep of processing, and iteratively refines these predictions by enhancing cell boundaries and clearing out spurious contours of elements like cell organelles (Fig. 3c).

heading: 5 BUGS OR BYPRODUCTS OF OPTIMIZED NEURAL COMPUTATIONS?

text:
Orientation-tilt illusion Like the neural circuit of Mély et al. (2018), fGRU modules are designed with an asymmetry in their ability to suppress and facilitate feedforward input (see Section 3). This
potentially gives γ-Nets (which contain fGRU modules) the capacity to exhibit similar contextual illusions as humans. Here, we tested whether a γ-Net trained on contour detection in natural images exhibits an orientation-tilt illusion.
We tested for this illusion by training orientation decoders on the outputs of models trained on the full BSDS500 dataset. These decoders were trained on 100K grating images, in which the center and surround orientations were the same (Fig. S2a). These images were sampled from all orientations and spatial frequencies. The decoders had two 1×1 convolution layers and an intervening linear rectification to map model outputs into the sine and cosine of grating orientation. Both the γ-Net and BDCN achieved nearly perfect performance on a held-out validation set of gratings.
We tested these models on 1K grating stimuli generated with different center-surround grating orientations (following the method of O’Toole & Wenderoth 1977, Fig. S2b), and recorded model predictions for the center pixel in these images (detailed in Appendix C). Surprisingly, γ-Net encodings of these test images exhibited a similar tilt illusion as found in human perceptual data (Fig. 4b). There was repulsion when the central and surround gratings had similar orientations, and attraction when these gratings were dissimilar. This illusory phenomenon cannot be explained by accidental factors such as aliasing between the center and the surround, which would predict the opposite pattern, suggesting that the illusion emerges from the model’s strategy for contour detection. In contrast, the BDCN, which only relies on feedforward processing, did not exhibit the effect (Fig. 4b). We also tested lesioned γ-Nets for the orientation-tilt illusion, but only the full γ-Net and a version lesioned to have only (spatially broad) horizontal connections replicated the entire illusion (thought this lesioned model performed worse on contour detection than the full model; Fig. S4). These findings are consistent with the work of Mély et al. (2018), who used spatially broad horizontal connections to model contextual illusions, as well as the more recent neurophysiological work of Chettih & Harvey (2019), who explained contextual effects through both horizontal and top-down interactions.
Correcting the orientation-tilt illusion What visual strategies does the orientation-tilt illusion reflect? We tested this question by taking a γ-Net that was trained to decode central grating orientation in tilt-illusion, and then training it further to decode the central grating orientation of tilt-illusion stimuli (Fig. 5a, “illusion-corrected” in red). Importantly, γ-Net weights were optimized during training, but the orientation decoder was not. Thus, improving performance for decoding the orientation of these illusory stimuli comes at the expense of changing γ-Net weights that were
responsible for its orientation-tilt illusion. As a control, another γ-Net was trained with the same routine to decode the orientation of full-image gratings, for which there is no illusion (Fig. 5a, “domain-transfer control” in blue; see Fig. S6 for training performance of both models). Both models were tested on the BSDS500 test set.
Correcting the orientation-tilt illusion of a γ-Net significantly hurts its object contour detection performance (Fig. 5a; 1-sample T-test of the per-image ODS F1 difference between models, T (199) = 13.570, p < 0.001). The illusion reflects γ-Net strategies for selecting object-boundaries rather than low-level contours (Fig. 5b; Fig. S7 for more examples).

heading: 6 CONCLUSION

text:
Why do we experience visual illusions? Our experiments indicate that one representative contextual illusion, the orientation-tilt illusion, is a consequence of neural strategies for efficient scene segmentation. We directly tested whether this contextual illusion is a bug or a byproduct of optimized neural computations using the γ-net: a dense prediction model with recurrent dynamics inspired by neural circuits in visual cortex. On separate contour detection tasks, the γ-Net performed on par with state-of-the-art models when trained in typical regimes with full augmented datasets, but was far more efficient than these models when trained on sample-limited versions of the same datasets. At the same time, the γ-Net exhibited an orientation-tilt illusion which biased it towards high-level object-boundary contours over low-level edges, and its performance was reduced when it was trained to correct its illusion.
While γ-Nets are more sample efficient than leading feedforward models for contour detection, they also take much more “wall-time” to train than these feedforward models. Learning algorithms and GPU optimizations for RNNs are lagging behind their feedforward counterparts in computational efficiency, raising the need for more efficient approaches for training hierarchical RNNs like γ-Nets to unlock their full potential.
More generally, our work demonstrates novel synergy between artificial vision and vision neuroscience: we demonstrated that circuit-level insights from biology can improve the sample efficiency of deep learning models. The neural circuit that inspired the fGRU module explained biological illusions in color, motion, and depth processing (Mély et al., 2018), and we suspect that γ-Nets will have similar success in learning sample-efficient strategies – and exhibiting contextual illusions – in these domains.

heading: B MEMBRANE PREDICTION MODELS

text:
Our reference model for membrane prediction is the 3D U-Net of (Lee et al., 2017). This architecture consists of four encoder blocks (multiple convolutions and skip connections, pooling and subsampling), followed by four decoder blocks (transpose convolution and convolution). This U-Net uses spatial pooling between each of its encoder blocks to downsample the input, and transpose convolutions between each of its decoder blocks to upsample intermediate activities. We validated our implementation of this U-Net following the author’s training routine, and were able to replicate their reported “superhuman” performance in cell segmentation on SNEMI3D (Fig. S1).
The γ-Net for connectomics resembles the U-Net architecture of Lee et al. (2017). This γ-Net replaces the blocks of convolutions and skip connections of that model with a single layer of convolution followed by an fGRU (as in the high level diagram of Fig. 1c, Conv(`),−→ fGRU(`)). In the encoder pathway, fGRUs store horizontal interactions between spatially neighboring units of the preceding convolutional layer in their hidden states. In the decoder pathway, γ-Net introduces fGRUs that learn top-down connections between layers, and connect recurrent units from higher-feature processing layers to lower-feature processing ones.
Name Tissue Imaging Resolution Voxels (X/Y/Z/Volumes) SNEMI3D Mouse cortex mbSEM 6× 6× 29nm 1024× 1024× 100× 1
Ding Mouse retina SBEM 13.2× 13.2× 26nm 384× 384× 384× 1
Table S3: SEM image volumes used in membrane prediction. SNEMI3D images and annotations are publicly available (Kasthuri et al., 2015), whereas the Ding dataset is a volume from (Ding et al., 2016) that we annotated.
Figure S2: Examples of tilt-illusion stimuli. (a) For training images, we sample over a range of size and wavelength to generate single oriented grating patches. (b) Test images are obtained by sampling a full range of surround orientation, while fixing all other parameters such as size and frequency of gratings as well as the orientation of the center gratings (at 45 degrees).
Key to the approach of Lee et al. (2017) is their use of a large set of random data augmentations applied to SEM image volumes, which simulate common noise and errors in SEM imaging. These are (i) misalignment between consecutive z-locations in each input image volume. (ii) Partial- or fully-missing sections of the input image volumes. (iii) Blurring of portions of the image volume. Augmentations that simulated these types of noise, as well as random flips over the xyz-plane, rotations by 90◦, brightness and contrast perturbations, were applied to volumes following the settings of Lee et al. (2017). The model was trained using Adam (Kingma & Ba, 2014) and the learning rate schedule of Lee et al. (2017), in which the optimizer step-size was halved when validation loss stopped decreasing (up to four times). Training involved single-SEM volume batches of 160 × 160 × 18 (X/Y/Z), normalized to [0, 1]. As in Lee et al. (2017), models were trained to predict nearest-neighbor voxel affinities, as well as 3 other mid- to long-range voxel distances. Only nearest neighbor affinities were used at test time.

heading: C ORIENTATION-TILT ILLUSION IMAGE DATASET

text:
Models were tested for a tilt illusion by first training on grating images of a single orientation, then testing on images in which a center grating had the same/different orientation as a surround grating. Each image in the training dataset consisted of a circular patch of oriented grating on a gray canvas of size 500 × 500 pixels. To ensure that the decoder successfully decoded orientation information from model activities, the training dataset incorporated a wide variety of grating stimuli with 4 randomly sampled image parameters: r, λ, θ, and φ. r denotes the radius of the circle in which oriented grating has been rendered, and was sampled from a uniform distribution with interval between 80 and 240 pixels; λ specifies the wavelength of the grating pattern and was sampled from a uniform distribution with interval between 30 and 90 pixels; θ specifies the orientation of the gratings and is uniformly sampled from all possible orientations; φ denotes the phase offset of the oriented gratings and is also uniformly sampled from all possible values. The models’ BSDS-trained weights were fixed and
Figure S3: Searching over learning rates did not rescue BSDS performance on small BSDS datasets had no affect on performance. over learning rates did not rescue from overfitting. The left panel depicts training and validation losses for BSDS on different sized subsets of BSDS500. (b) Performance after training with three different learning rates on the 5% split. There is little difference in best validation performance between the three learning rates. (c) The full training and validation loss curves for the BDCN trained on 5% of BSDS. The model overfits immediately. The model also overfit on the other dataset sizes, but because there was more data, this happened later in training.
readout layers were trained to decode orientation at the center of each image (procedure described in the main text).
This setup allowed us to tease apart the effects of the surround on the representation of orientation in the center by introducing separate surround regions in each test image filled with gratings with same/different orientations as the center (Fig.S2b). Each test image was generated with one additional parameter, ∆θ which specified orientation difference of the surround gratings with respect to the center orientation, θ, and was sampled from a uniform distribution with interval between −90 and +90 degrees. The radius of the surround grating is denoted by r and was sampled from the same uniform distribution we used in training dataset. Center gratings are then rendered in a circle of radius that is one half of the surround gratings.
Figure S4: Performance of lesioned γ-Nets. We evaluate the F1 ODS of these models on BSDS500 test images (left) and test for the presence of an orientation-tilt illusion (right). Top row: A comparison of γ-Nets with “untied weights” (i.e., unrolled to 8-timesteps with unique weights on every timestep), only top-down connections, only horizontal connections (using the standard 3×3 horizontal kernels), and only horizontal connections but with larger kernels (15×15). The full γNet outperformed each of these models on all subsets of BSDS500 data. Both horizontal-only models captured the repulsive regime of the orientation-tilt illusion when center and surround gratings were similar, but only the version with larger kernels also showed an attractive regime, when the center and surround orientations were dissimilar. Bottom row: A comparison of γ-Nets with no recurrence (i.e., one timestep of processing), no constraint for non-negativity (see fGRU formulation in main text), no parameters for additive feedback (µ and κ), and no parameters for multiplicative feedback (γ and ω). Once again, the full γ-Net outperformed each of these models. While none of these models showed the full orientation-tilt illusion, the γ-Net without non-negativity constraints and the γ-Net without additive feedback showed the repulsive regime of the orientation-tilt illusion.
PredictionImage Label Prediction per processing timestep1 8
Ti m
es te
p di ffe
re nc
e (L
2)
2 3 4 5 6 7 8 Timesteps
.05
.10
.15
.20
.25
.30
.35
Figure S5: γ-Nets trained for contour detection learn to approach a steady-state solution. The processing timecourse of γ-Net predictions on representative images from the BSDS500 test set. The L2 norm of per-pixel differences between predictions on consecutive timesteps (i.e., timestep 2 - timestep 1) approaches 0, indicating that the model converges towards steady state by the end of its processing timecourse.
Recurrent connections finetuned on:
Undoing an emergent tilt-illusion worsens ɣ-Net contour detection
Domain-transfer Control Illusion-corrected
Steps of training
G ra
tin gs
L 2
lo ss
F1
O D
S
(a) (b)
1 50 100 150 200 250 300 300 0.0
0.2
0.4
0.6 0.5
0.7
0.8
0.9
0.6
La be l La be l
Steps of training 1 80 160 240
Figure S6: Performance of γ-Nets during experiments to correct an orientation-tilt illusion. The illusion-corrected model was trained to have veridical representations of the central grating in tilt-illusion stimuli. To control for potential detrimental effects of the training procedure per se, a control model (“domain-transfer control”) was trained to decode orientations of single-grating stimuli. (a) Training causes contour-detection performance of both models to drop. However, the illusion-corrected model performance drops significantly more than the biased model (see main text for hypothesis testing). The losses for both models converge towards 0 across training, indicating that both learned to decode central-orientations of their stimuli. (b) Contour detection examples for biased and bias-corrected models across steps of this training procedure.
(b)
Im ag e La be l
Di ffe
re nc
e
Illusion-corrected ɣ-Net Domain-transfer control ɣ-Net Feature preferred by:
Figure S7: Differences in contour predictions for the illusion-corrected and domain-transfer control γ-Nets on BSDS500.

title: RECURRENT NEURAL CIRCUITS FOR CONTOUR DETECTION


summaries: This paper introduces a context-aware neural network (conCNN) that integrates context semantics into account for object detection. The proposed approach achieves this by embedding a context-aware module into the Faster R-CNN detection framework. The context-aware module simulates the learning process of Conditional Random Fields (CRF) model using a stack of common CNN operations. Specifically, this paper employs the mean-field approach of [1]. Experiments are performed on COCO dataset.

summaries: The paper proposes a contextual reasoning module following the approach proposed by the NIPS 2011 paper for object detection. Specifically, the algorithm proposed by NIPS 2011 are first converted to end-to-end modules step-by-step, and then added to the detection framework Faster R-CNN. The comparison is only to one other baseline (Relation Module), and some improvements are show especially for small objects.

summaries: This paper proposes a CRF-based context module for CNN-based object detectors. In particular for the two-stage region-based detector, like Faster RCNN, the context module is added right before the output layer of the classification head. Every box proposed by the RPN is a node in the CRF, and its label is the classification label. Message passing is unrolled as neural network layers. Potentials are defined based on object detector outputs, box overlap, and co-occurrence of class labels. Experiments are performed on the MS COCO object detection task.

abstractText: Although the state-of-the-art object detection methods are successful in detecting and classifying objects by leveraging deep convolutional neural networks (CNNs), these methods overlook the semantic context which implies the probabilities that different classes of objects occur jointly. In this work, we propose a context-aware CNN (or conCNN for short) that for the first time effectively enforces the semantics context constraints in the CNN-based object detector by leveraging the popular conditional random field (CRF) model in CNN. In particular, conCNN features a context-aware module that naturally models the mean-field inference method for CRF using a stack of common CNN operations. It can be seamlessly plugged into any existing region-based object detection paradigm. Our experiments using COCO datasets showcase that conCNN improves the average precision (AP) of object detection by 2 percentage points, while only introducing negligible extra training overheads.

authors:
Jifeng Dai
Yi Li
Kaiming He
Jian Sun

title: R-FCN: object detection via region-based fully convolutional networks

venue: In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems

authors:
Santosh Kumar Divvala
Derek Hoiem
James Hays
Alexei A. Efros
Martial Hebert

title: An empirical study of context in object detection

venue: In CVPR,

authors:
Pedro F. Felzenszwalb
Ross B. Girshick
David A. McAllester
Deva Ramanan

title: Object detection with discriminatively trained part-based models

venue: IEEE Trans. Pattern Anal. Mach. Intell.,

authors:
Carolina Galleguillos
Andrew Rabinovich
Serge J. Belongie

title: Object categorization using co-occurrence, location and appearance

title: Fast r-cnn

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Ross Girshick
Jeff Donahue
Trevor Darrell
Jitendra Malik

title: Rich feature hierarchies for accurate object detection and semantic segmentation

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Ian Goodfellow
Yoshua Bengio
Aaron Courville

title: Deep Learning

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Spatial pyramid pooling in deep convolutional networks for visual recognition

authors:
Kaiming He
Georgia Gkioxari
Piotr Dollár
Ross Girshick

title: Mask r-cnn

authors:
Han Hu
Jiayuan Gu
Zheng Zhang
Jifeng Dai
Yichen Wei

title: Relation networks for object detection

venue: In CVPR, Salt Lake City, UT,

authors:
Philipp Krähenbühl
Vladlen Koltun

title: Efficient inference in fully connected crfs with gaussian edge potentials

venue: In NIPS,

authors:
Lubor Ladicky
Chris Russell
Pushmeet Kohli
Philip H.S. Torr

title: Associative hierarchical crfs for object class image segmentation

venue: In ICCV, Kyoto, Japan, September 27 - October

authors:
John D. Lafferty
Andrew McCallum
Fernando C.N. Pereira

title: Conditional random fields: Probabilistic models for segmenting and labeling sequence data

authors:
Tsung-Yi Lin
Piotr Dollár
Ross Girshick
Kaiming He
Bharath Hariharan
Serge Belongie

title: Feature pyramid networks for object detection

authors:
Roozbeh Mottaghi
Xianjie Chen
Xiaobai Liu
Nam-Gyu Cho
Seong-Whan Lee
Sanja Fidler
Raquel Urtasun
Alan L. Yuille

title: The role of context for object detection and semantic segmentation in the wild

authors:
Andrew Rabinovich
Andrea Vedaldi
Carolina Galleguillos
Eric Wiewiora
Serge J. Belongie

title: Objects in context

authors:
Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster r-cnn: Towards realtime object detection with region proposal networks. In Advances in neural information processing

authors:
Antonio Torralba
Kevin P. Murphy
William T. Freeman
Mark A. Rubin

title: Context-based vision system for place and object recognition

authors:
Zhuowen Tu
Xiang Bai

title: Auto-context and its application to high-level vision tasks and 3d brain image segmentation

authors:
Shuai Zheng
Sadeep Jayasumana
Bernardino Romera-Paredes
Vibhav Vineet
Zhizhong Su
Dalong Du
Chang Huang
Philip H.S. Torr

title: Conditional random fields as recurrent neural networks

heading: 1 Introduction

text:
In recent years, deep convolutional neural networks (CNN) (Goodfellow et al., 2016) have been used with great success in object detection tasks. However, some well known principles in computer vision that have been shown to be effective in object detection are largely overlooked. In particular, the vision community has shown that the semantic context, namely the correlations among the objects, helps object detection Rabinovich et al. (2007). That is, when performing the task of object detection, objects’ class labels should be inferred with respect to other objects in the scene. For example, when detecting objects in a scene of a baseball match in the COCO dataset, which contains four objects: “Baseball”, “Baseball bat”, “human”, and “baseball glove”, the CNN-based object detection model tends to recognize the “baseball bat” as “tooth brush”. Yet it can easily be seen that this object of “tooth brush” does not fit into the context with other labels. This is because the state-of-the-art object detection methods (He et al., 2015; Girshick, 2015; Ren et al., 2015; Dai et al., 2016; Lin et al., 2017; He et al., 2017) adopt the region-based paradigm since it was introduced in the seminal R-CNN work (Girshick et al., 2014). Given a set of region proposals, these methods perform object classification and bounding box regression on each proposal individually, without taking the semantic context into consideration.
In this work, we now propose a context-aware object detection strategy called conCNN to address the above shortcoming. By seamlessly plugging a context-aware module into the existing CNN-based object detection paradigm, conCNN automatically learns and effectively enforces the semantics context constraints, yet requiring minimal modification of the existing deep object detection architecture.
conCNN is inspired by our observation that probabilistic graphical models, in particular, Conditional Random Fields (CRFs) have been successful in enhancing the accuracy of low level computer vision tasks (Ladicky et al., 2009; Zheng et al., 2015; Rabinovich et al., 2007; Krähenbühl & Koltun, 2011) such as image segmentation or object detection. More specifically, CRF models the label assignment problem as a probabilistic inference problem such that constraints can be incorporated, for example, the label agreement between similar pixels in image segmentation.
We thus design a context-aware module that for the first time leverages CRF in deep object detection architecture to maximize the contextual agreement among the objects in the same scene, yet only slightly increasing the training time of the object detection model. The overall context-aware deep object detection network, corresponds to a deep architecture with a traditional CNN-based module concatenated by a context-aware module, can be trained end-to-end following the common practice of back-propagation in deep neural networks. Therefore, we successfully combine the strengths of both CNN and CRF-based graphical models in one unified framework.
More specifically, the context-aware module reformulates the iterative mean-field inference (Krähenbühl & Koltun, 2011) – a popular approximate inference method for CRF as a stack of common CNN layers. This way, it learns a compatibility matrix as normal parameters of deep neural network that represents the probabilities of label co-occurrences, as the compatibility function in CRF.
Our experimental evaluation on the COCO datasets confirms that conCNN improves the AP of object detection by 2 percentage points.

heading: 2 Network Architecture of conCNN

text:
In this work, we proposal a context-aware neural network (conCNN) for object detection. Essentially, we achieve this by plugging a context-aware module into a CNN network built upon the popular region-based object detection paradigm as shown in Fig. 1.
Region-based Object Detection. The region-based object detection paradigm (Girshick et al., 2014) first processes the whole image with several convolutional and max pooling layers (the CNN backbone) to produce a conv feature map. Taking the feature map as input, the Region Proposal Network (RPN in short) generates a set of rectangular object proposals that most likely contain objects. Then for each object proposal, a region of interest (RoI) pooling layer extracts a fixed-length feature vector from the feature map.
Each feature vector is fed into a sequence of fully connected layers (FC layers) that finally branch into two sibling output layers corresponding to the classification task and the bounding box regression task. The classification task produces softmax probability estimates over L object classes. The bounding box regression task outputs four real-valued numbers for each of the L object classes. Each set of 4 values encodes refined bounding box positions for one of the L classes.
Context-aware Module. Our context-aware module is plugged into the classification task. Specifically, it takes the prediction from the last FC layer in the classification task as well as the location for each bounding box in the regression task as input and produces a modified prediction that incorporates the context semantics. The modified prediction distribution is then fed into the softmax layer to generate the final probability estimates.

heading: 3 Conditional Random Field in Object Detection

text:
In this section we briefly overview Conditional Random Fields (CRF) for object detection. In object detection, a CRF models object labels (classes) as random variables that form a Markov Random Field (MRF) when conditioned upon the input image.
Conditional Random Field. Let Xi be the random variable w.r.t. object i. It represents the label assigned to the object i and can take any value from a pre-defined set of labels L = {l1 , l2 , ..., lL}. Let X be the vector formed by the random variables X1, X2, ..., XN , where N denotes the number of objects in the image I. Then the probability of some label assignment x conditional on the global observation I is given by the CRF model:
P (X = x|I) = 1 Z(I) exp(−E(x)) (1)
Here E (x ) denotes the energy of the configuration x ∈ LN , while Z (x ) denotes the partition function (Lafferty et al., 2001).

heading: 3.1 Mean-field Approximation

text:
Minimizing the above CRF energy E(x) yields the most probable label assignment x (maximizing P(x )) for the given image. Since this exact minimization is intractable, in the fully connected pairwise CRF model, a mean-field approximation to the CRF distribution (Krähenbühl & Koltun, 2011) is used for approximate maximum posterior marginal inference. It approximates the CRF distribution P(X) using a simpler distribution Q(X), which can be expressed as the product of independent marginal distributions, Q(X) = ∏N i Qi(Xi), where Qi(Xi) corresponds to one object i in image I. Leveraging Krähenbühl & Koltun (2011) for object detection, we use the following update equation to iteratively compute a valid Qi(Xi):
Qi(Xi = l) = 1
Z(I) exp  −ψu(Xi = l)︸ ︷︷ ︸ Unary Component − ∑ l′∈L µ(l, l′) K∑ m=1 wm ∑ i6=j km(fi, fj)Qj(Xj = l ′)︸ ︷︷ ︸
Pairwise Component
 (2)
In Equation 2, the unary component −ψu(Xi = l) (denoted as Ui(l)) measures the probability that the label l is assigned to the object i (Xi = l). As shown in Fig. 1, in our conCNN it is obtained from the classification task, which assigns label to object i without considering the context semantics.
The pairwise component ∑ l′∈L µ(l , l ′) ∑K m=1 w m ∑ i 6=j k m(fi , fj )Qj (xj = l
′) reflects the pairwise influence of the label assignment (Xj = l′) of objects j on the label assignment of object i (Xi = l). Generally speaking, The pairwise component measures the cost of assigning labels l , l′ to objects i, j simultaneously. It enforces the context semantics constraints.
Among this pairwise component, function µ(l, l′) captures the compatibility (or the possibility of co-occurrence) between a pair of labels l and l′. It corresponds to the new parameters that have to be learned in the training of conCNN to enforce the context semantics constraints.
Each function km(fi , fj ) for m = 1, ..., M, represents a function applied on feature vectors (fi, fj) of objects i and j. The feature vector fi of object i corresponds to some characteristics of object i derived from a CNN. In our scenarios, fi corresponds to the location of the bounding box or the probability estimation of object i taking label l ∈ L. Essentially, km(fi , fj ) reflects how strongly an object i is related to other objects j, hence called relation function.
Note in Krähenbühl & Koltun (2011) and Zheng et al. (2015) which apply CRF in image segmentation, km(fi , fj ) corresponds to the Gaussian kernel. However, in object detection Gaussian kernel does not necessarily work well (Rabinovich et al., 2007). In Sec. 4.2, we will introduce in more details the relation functions used in our work.
Algorithm 1 Mean-field for dense pairwise CRFs
1: Qi(l) ← 1Z (I )exp(Ui(l)); . Initialization 2: while not converged do 3: Q̂mi (l) ← ∑ j 6=i k
m(fi , fj )Qj (l); . Message passing from all Xj to all Xi 4: Q̂i(l) ← ∑ m w
mQ̂mi (l); . Weighting 5: Q̂i(l) ← ∑ l′∈L µ(l, l ′)Q̂i(l ′); . Compatibility transform 6: Q̂i(l) ← Ui(l) - Q̂i(l); . Local Update 7: Qi(l) ← 1Z (I )exp(Q̂i(l)); . Normalization 8: end while

heading: 3.2 Mean-field Algorithm

text:
The update equation (Equation 2) leads to the mean-field algorithm summarized in Algorithm 1 (Krähenbühl & Koltun, 2011). Each iteration of Algorithm 1 performs a message passing step, a compatibility transform, a local update, and a normalization.
Next, we show how to reformulate all these steps in Algorithm 1 as common CNN layers and hence effectively enforces the context semantics constraints in CNN-based object detection, using the classical practice of back propagation in deep neural networks.

heading: 4 CRF Inference As Common CNN Layers

text: To reformulate the mean-field inference as common CNN layers, it is essential to be able to calculate error differentials in each layer w.r.t. its inputs in order to back-propagate the error differentials to previous layers during training. This way, CRF parameters such as the label compatibility function µ(l , l ′) can be learned automatically during the training of the full network. Therefore, in this section we also discuss how to calculate error differentials with respect to the parameters in each layer.

heading: 4.1 Initialization

text:
In this step (Line 1, Algorithm 1), the initialization operation Qi(l) = 1Z (I )exp(Ui(l)) is performed. Since Z (I ) (Lafferty et al., 2001) corresponds to ∑ l∈L exp(Ui(l)), we get:
Qi(l) = 1
Z(I) exp(Ui(l)) = exp(Ui(l))∑ l∈L exp(Ui(l))
(3)
Note that the form of Qi(l) in Equation 3 is equivalent to the softmax function (Goodfellow et al., 2016), widely used in CNN architectures. Naturally it supports back-propagation. That is, during back-propagation its error differentials received at the output of this step is passed back to the input of the context-aware module (the prediction produced by bounding box classification) after performing backward pass calculations of the softmax function.

heading: 4.2 Message Passing

text:
The message passing step updates each single variable Xi w.r.t. object i by aggregating information from all other variables Xj w.r.t. other objects j. As shown in Fig. 2, conCNN supports two types of information aggregation, called local message passing and global message passing.
Local Message Passing. Local message passing models the geometry relationship among the objects in the same image. Similar to Hu et al. (2018), it is based on the intuition that given an object i, the objects j tend to have strong influence on i if its bounding box overlaps with the bounding box of i. Therefore, the local message passing takes the location information into consideration when aggregating the influence from other objects.
More specifically, we first compute the Intersect of Union (IoU) between the bounding boxes of a pair of objects. The IoUs w.r.t. all object pairs correspond to a N ×N matrix, where N denotes the number of objects in image I. To compute the IoU matrix, we need as input the bounding box information of all N objects which can be considered as a N × 4 matrix. To efficiently compute the IoU matrix, in this work we formulate its computation as a matrix transformation operation such that this process can be performed on GPU in parallel. The diagonals of the IoU matrix are then reset to zeros because we only need to aggregate the predictions from other objects.
Once the IoU matrix is ready, the local message passing process can be mapped to a matrix product operation between the N ×N IoU matrix and a N × L prediction matrix which correspond to the original predictions w.r.t. the N objects and the L classes. This results in the final aggregated predictions (a N × L matrix), weighting the predictions from the overlapping bounding boxes more.
Since this step is equivalent to a set of linear weighted-sum transformations and does not involve any parameters, the back-propagation is trivially supported.
Global Message Passing. Inherent co-occurrence relations also exist among the objects in the same scene, not tied to the geometry locations of these objects. For instance, although keyboard and mouse do not overlap, the two object classes together define a computer-related context and can help differentiate mouse and soap which look similar in appearance.
In this work, this o-occurrence relationship is captured by the global message passing. It uses the probability that each object class l appears in an image to model the context of this image, denoted as p(l). Here we define p(l) as the maximal probability estimation of class l among all objects in the image, that is:
p(l) = max{pi(l)|∀object i ∈ I} (4)
where pi(l) denotes the probability that the classification task assigns object i to class l.
The p(l) w.r.t. all l ∈ L can be computed by applying a global max pooling function (Goodfellow et al., 2016) on the N × L prediction matrix. This produces a 1× L class probability vector, which as the global context of this image, influences all objects equally.
Since max pooling is a common operation in CNN, the back-propagation can be naturally supported in the typical way.

heading: 4.3 Weighting Step

text: The weighting step takes a weighted sum of the M outputs of the message passing step, corresponding to the M relation functions. This step can be mapped to a 1× 1 convolution filter with M input channels and 1 output channel. Here each input channel corresponds to a N × L matrix. The 1× 1×M × 1 parameters of this filter correspond to the weights wm of the M relation functions. As a common convolution operation, error derivative can be computed in the usual manner to pass the error derivatives back to the previous step.

heading: 4.4 Compatibility Transform

text:
The outputs Q̂i(l) on different labels l ∈ L influence each other to a variant extent, depending on the compatibility between these labels. In conCNN, ∀ two labels l and l′ ∈ L, the compatibility is parameterized by the compatibility function µ(l, l′), corresponding to a L×L compatibility matrix C.
Organizing Q̂i(l) with respect to all N objects in image I and all labels l ∈ L as a N × L matrix Q̂, the compatibility transform step can be implemented as a matrix product: Q̂ × C. The output is also a N × L matrix. In CNN, this matrix product can be viewed as a common convolution filter, where the spatial receptive field of the filter is 1× 1, and the number of input and output channels are both L. The 1× 1× L× L weights of this filter correspond to the compatibility matrix C. In other words, learning the label compatibility function µ is equivalent to learning the weights of this filter. Since this step is a common convolution operation, transferring error differentials from the output to its input can be done in the usual way.

heading: 4.5 Local Update

text: In this step (Line 6, Algorithm 1), the output Q̂i(l) from the compatibility transform step is subtracted from the unary input Ui(l). Since this step does not have any parameters, transferring error differentials can be done trivially by copying the differentials at the output of this step to both inputs Q̂i(l) and Ui(l).

heading: 4.6 Normalization

text: Similar to the initialization step, the normalization step (Line 7, Algorithm 1) corresponds to a softmax operation without parameters.

heading: 5 Experimental Evaluation

heading: 5.1 Overview of Experimental Setting

text:
Datasets. We demonstrate the effectiveness of our context-aware module using the benchmark COCO dataset and a subset of it.
The original COCO dataset contains 80 categories. As in previous work (He et al., 2017; Lin et al., 2017), we train using the union of 80k train images and a 35k subset of validation images (trainval35k), and evaluate on the remaining 5k validation images (minival).
The COCO subset dataset contains 6 categories of the original COCO including baseball bat, baseball glove, sink, toilet, mouse and keyboard. These 6 categories represent 3 different scenes (outdoor, bathroom, indoor) with 2 classes from each scene. There are 13k training images and 531 validation images in this COCO subset.
Methodology. We evaluate: (1) Faster R-CNN (Lin et al., 2017) as baseline; (2) Relation Network: Faster R-CNN with Relation Module (Hu et al., 2018); (3) Our method conCNN: Faster R-CNN with our context-aware module (Sec. 2). We ran experiments on 4 P100 GPU instances on Google cloud. The results show that our conCNN outperforms Relation Network in improving the AP of object recognition.
Evaluation Metrics. We report the standard COCO metrics including AP (averaged over IoU thresholds from 0.5 to 0.95), AP50 (IoU = 0.5), AP75 (IoU = 0.75) and AP at different scales including APS (Small), APM (Medium), APL (Large).

heading: 5.2 Implementation Details

text:
All approaches use ResNet-101-FPN as backbone. We implemented our conCNN by plugging the context-aware module into the penultimate layer of the classification task. We set
hyper-parameters following the existing Faster R-CNN work (Lin et al., 2017; Ren et al., 2015). The evaluation of the baseline (Lin et al., 2017) and Relation Network (Hu et al., 2018) is based on the models published by the authors of Relation Network.
During training, we adopt image-centric training (Girshick, 2015). Images are resized such that their scale (shorter edge) is 800 pixels as done in (Lin et al., 2017). An RoI is considered positive if it has IoU with a ground-truth box of at least 0.5 and negative otherwise. Each mini-batch has 4 images per GPU and each image has 512 sampled RoIs, with a ratio of 1:3 of positives to negatives. For the COCO subset with 6 categories, we train on 4 GPUs for 12k iterations, with a learning rate of 0.02 which is decreased by 10 at the 9kth iteration. For the COCO dataset with 80 categories, we train on 4 GPUs for 90k iterations, with a learning rate of 0.02 which is decreased by 10 at the 60kth and 80kth iteration. At test time, the proposal number is 1000 (as in Lin et al. (2017)).

heading: 5.3 Experiments on COCO subset with 6 categories

text:
We first test our conCNN on the COCO subset with 6 categories. This subset contains images from 3 different scenes including outdoor, bathroom, and indoor. We use this subset to validate that our conCNN can effectively capture the co-occurrence relationship between the objects in the same scene.
As shown in Table 1, in almost all cases, our conCNN outperforms Faster R-CNN and Relation Network. The performance gain results from the context-aware module in our conCNN, which leverages both the geometry and co-occurrence relationships among the objects. In particular, conCNN is good at detecting small and medium objects. Although small and medium objects are difficult to recognize purely by appearance, the relationships among the objects can help emphasize the correlated objects in the same image and thus improve the APS and APM . Relation network does not perform well on this dataset because the Relation Network only considers the geometry relationship among the objects overlapping with each other, while objects hardly overlap in this dataset.
In addition, our conCNN introduces negligible training overhead. Specifically, the training takes 4.12 hours and 4.24 hours for Faster R-CNN and conCNN respectively. Relation Network takes 5.02 hours for training, because it uses a more complex relation module.
Fig. 3 shows the compatibility matrix learnt by conCNN. By Eq. 2, correlated classes should have negative values in this matrix. The smaller the value is, the stronger the correlation between the pair of classes. As expected, the learnt compatibility matrix shows that baseball glove is strongly correlated to baseball bat, while toilet/mouse is highly correlated to sink/keyboard. This confirms the effectiveness of conCNN in capturing context semantics.

heading: 5.4 Experiments on COCO with 80 categories

text: We also test our conCNN on the whole COCO dataset with 80 categories. As shown in Table 2, our conCNN still outperforms Faster R-CNN and Relation Network. Relation network performs well when detecting large objects. That is as expected because the Relation Network only considers the geometry relationship among the objects overlapping with each other, while large objects tend to overlap with more objects than small objects. The training time of conCNN is 27.25 hours – only 0.75 hours longer than the Faster R-CNN which takes 26.5 hours. On the other hand, the Relation Network takes 32.25 hours to train.

heading: 6 Related Work

text:
CRF in Computer Vision. Zheng et al. (2015) combines CRF and CNN to perform pixel-level semantic image segmentation. It formulates the CRF as an RNN. The RNN is then plugged in as a part of a CNN which can be trained end-to-end using back-propagation. Our work instead focuses on leveraging CRF in object detection. Instead of modeling the training process of CRF as a RNN, we factorize the mean-field approximate inference of CRF as common CNN layers and use the common CNN operations such as softmax and convolution to naturally simulate CRF in enforcing the context semantics constraints. Further, unlike Zheng et al. (2015) which uses Gaussian kernels to encourage label agreement between similar pixels, we design customized functions for object detection task. These functions effectively reflect the co-occurrence relationship among the objects in the same scene. CRF was also used as post-processing to refine the pixel-level label predictions or object-level class assignment (Rabinovich et al., 2007; Ladicky et al., 2009; Krähenbühl & Koltun, 2011). Since CRF is not incorporated into the training process of segmentation or object detection model, these methods are shown to be not as effective as Zheng et al. (2015).
Object Relation in Deep Learning. Hu et al. (2018) proposed a relation network that adapts the concept of attention in natural language processing (NLP) into the classical CNN-based object detection framework. The key idea is to use attention to model the geometry relationships among the objects in the same image, such as a plate on top of a table or a tooth brush in a cup. These dependencies are represented as relation features which are used together with other features to classify objects. Our work instead combines the strengths of CNN and CRF. It not only models the geometry relationships among the objects in one image, but also leverages the inherent co-occurrence relationships between object classes, such as mouse and keyboard, fridge and microwave, or toilet and sink.
Object Relation in Classical Computation Vision. Similar to the use of CRF, most object detection works before deep learning arises use object relations as a post-processing step (Divvala et al., 2009; Galleguillos et al., 2008; Mottaghi et al., 2014; Torralba et al., 2003; Tu & Bai, 2010; Rabinovich et al., 2007; Felzenszwalb et al., 2010). The detected objects are re-scored by considering object relationship such as the co-occurrence of different object classes (Rabinovich et al., 2007; Felzenszwalb et al., 2010). However, unlike our work, in this setup the object detector is unaware of the object relations during the training phase. Therefore, it cannot fully employ the object relations in object detection.

heading: 7 Conclusion

text: In this work, we proposed a context-aware neural network conCNN that takes the context semantics into consideration in object detection. The key idea is to embed a context-aware module into the CNN-based object detection network that effectively simulates the learning process of Conditional Random Field (CRF) model using a stack of common CNN operations. Combining the strengths of both CNN and CRF, conCNN effectively improve the AP of object detection as confirmed in our experiments on COCO datasets.

title: Context-Aware Object Detection With Con- volutional Neural Networks