Sum-product networks (SPNs) are an expressive class of deep probabilistic models in which inference takes time linear in their size, enabling them to be learned effectively. However, for certain challenging problems, such as scene understanding, the corresponding SPN has exponential size and is thus intractable. In this work, we introduce submodular sum-product networks (SSPNs), an extension of SPNs in which sum-node weights are defined by a submodular energy function. SSPNs combine the expressivity and depth of SPNs with the ability to efficiently compute the MAP state of a combinatorial number of labelings afforded by submodular energies. SSPNs for scene understanding can be understood as representing all possible parses of an image over arbitrary region shapes with respect to an image grammar. Despite this complexity, we develop an efficient and convergent algorithm based on graph cuts for computing the (approximate) MAP state of an SSPN, greatly increasing the expressivity of the SPN model class. Empirically, we show exponential improvements in parsing time compared to traditional inference algorithms such as α-expansion and belief propagation, while returning comparable minima.
1 INTRODUCTION
Sum-product networks (SPNs) (Poon & Domingos, 2011; Gens & Domingos, 2012) are a class of deep probabilistic models that consist of many layers of hidden variables and can have unbounded treewidth. Despite this depth and corresponding expressivity, exact inference in SPNs is guaranteed to take time linear in their size, allowing their structure and parameters to be learned effectively from data. However, there are still many models for which the corresponding SPN has size exponential in the number of variables and is thus intractable. For example, in scene understanding (or semantic segmentation), the goal is to label each pixel of an image with its semantic class, which requires simultaneously detecting, segmenting, and recognizing each object in the scene. Even the simplest SPN for scene understanding is intractable, as it must represent the exponentially large set of segmentations of the image into its constituent objects.
Scene understanding is commonly formulated as a flat Markov (or conditional) random field (MRF) over the pixels or superpixels of an image (e.g., Shotton et al. (2006); Gould et al. (2009)). Inference in MRFs is intractable in general; however, there exist restrictions of the MRF that enable tractable inference. For pairwise binary MRFs, if the energy of each pairwise term is submodular (alternatively, attractive or regular) (Kolmogorov & Zabih, 2004), meaning that each pair of neighboring pixels prefers to have the same label, then the exact MAP labeling of the MRF can be recovered in low-order polynomial time through the use of a graph cut algorithm1 (Greig et al., 1989; Boykov & Kolmogorov, 2004). This result from the binary case has been used to develop a number of powerful approximate algorithms for the multi-label case (e.g., Komodakis et al. (2007); Lempitsky et al. (2010)), the most well-known of which is α-expansion (Boykov et al., 2001), which efficiently returns an approximate labeling that is within a constant factor of the true optimum by solving a series of binary graph cut problems. Unfortunately, pairwise MRFs are insufficiently expressive for com-
1Formally, a min-cut/max-flow algorithm(Ahuja et al., 1993) on a graph constructed from the MRF.
plex tasks such as scene understanding, as they are unable to model high-level relationships, such as constituency (part-subpart) or subcategorization (superclass-subclass), between arbitrary regions of the image, unless these can be encoded in the labels of the MRF and enforced between pairs of (super)pixels. However, this encoding requires a combinatorial number of labels, which is intractable. Instead, higher-level structure is needed to efficiently represent these relationships.
In this paper, we present submodular sum-product networks (SSPNs), a novel model that combines the expressive power of sum-product networks with the tractable segmentation properties of submodular energies. An SSPN is a sum-product network in which the weight of each child of a sum node corresponds to the energy of a particular labeling of a submodular energy function. Equivalently, an SSPN over an image corresponds to an instantiation of all possible parse trees of that image with respect to a given image grammar, where the probability distribution over the segmentations of a production on a particular region is defined by a submodular random field over the pixels in that region. Importantly, SSPNs permit objects and regions to take arbitrary shapes, instead of restricting the set of possible shapes as has previously been necessary for tractable inference. By exploiting submodularity, we develop a highly-efficient approximate inference algorithm, INFERSSPN, for computing the MAP state of the SSPN (equivalently, the optimal parse of the image). INFERSSPN is an iterative move-making-style algorithm that provably converges to a local minimum of the energy, reduces to α-expansion in the case of a trivial grammar, and has complexity O(|G|c(n)) for each iteration, where c(n) is the complexity of a single graph cut and |G| is the size of the grammar. As with other move-making algorithms, INFERSSPN converges to a local minimum with respect to an exponentially-large set of neighbors, overcoming many of the main issues of local minima (Boykov et al., 2001). Empirically, we compare INFERSSPN to belief propagation (BP) on a multilevel MRF and to α-expansion on an equivalent flat MRF. We show that INFERSSPN parses images in exponentially less time than both of these while returning energies comparable to α-expansion, which is guaranteed to return energies within a constant factor of the true optimum.
The literature on using higher-level information for scene understanding is vast. We briefly discuss the most relevant work on hierarchical random fields over multiple labels, image grammars for segmentation, and neural parsing methods. Hierarchical random field models (e.g., Russell et al. (2010); Lempitsky et al. (2011)) define MRFs with multiple layers of hidden variables and then perform inference, often using graph cuts to efficiently extract the MAP solution. However, these models are typically restricted to just a few layers and to pre-computed segmentations of the image, and thus do not allow arbitrary region shapes. In addition, they require a combinatorial number of labels to encode complex grammar structures. Previous grammar-based methods for scene understanding, such as Zhu & Mumford (2006) and Zhao & Zhu (2011), have used MRFs with AND-OR graphs (Dechter & Mateescu, 2007), but needed to restrict their grammars to a very limited set of productions and region shapes in order to perform inference in reasonable time, and are thus much less expressive than SSPNs. Finally, neural parsing methods such as those in Socher et al. (2011) and Sharma et al. (2014) use recursive neural network architectures over superpixel-based features to segment an image; thus, these methods also do not allow arbitrary region shapes. Further, Socher et al. (2011) greedily combine regions to form parse trees, while (Sharma et al., 2014) use randomly generated parse trees, whereas inference in SSPNs finds the (approximately) optimal parse tree.
2 SUBMODULAR SUM-PRODUCT NETWORKS
In the following, we define submodular sum-product networks (SSPNs) in terms of an image grammar because this simplifies the exposition with respect to the structure of the sum-product network (SPN) and because scene understanding is the domain we use to evaluate SSPNs. However, it is not necessary to define SSPNs in this way, and our results extend to any SPN with sum-node weights defined by a random field with submodular potentials. Due to lack of space we refer readers to Gens & Domingos (2012), Poon & Domingos (2011) and Gens & Domingos (2013) for SPN details.
With respect to scene understanding, an SSPN defines a generative model of an image and a hierarchy of regions within that image where each region is labeled with a production (and implicitly by the head symbol of that production), can have arbitrary shape, and is a subset of the region of each of its ancestors. An example of an SSPN for parsing a farm scene is shown in Figure 1. Given a starting symbol and the region containing the entire image, the generative process is to first choose a production of that symbol into its constituent symbols and then choose a segmentation of the region into a set of mutually exclusive and exhaustive subregions, with one subregion per constituent sym-
bol. The process then recurses, choosing a production and a segmentation for each subregion given its symbol. The recursion terminates when one of the constituents is a terminal symbol, at which point the pixels corresponding to that region of the image are generated. This produces a parse tree in which each internal node is a pair containing a region and a production of the region, and the leaves are regions of pixels. For each node in a parse tree, the regions of its children are mutually exclusive and exhaustive with respect to the parent node’s region. As in a probabilistic context-free grammar (PCFG) (Jurafsky & Martin, 2000), productions are chosen from a categorical distribution over the productions of the current symbol. Segmentations of a given region, however, are sampled from a (submodular) Markov random field (MRF) over the pixels in the region.
Formally, let G = (N,Σ, R, S,w) be a non-recursive stochastic grammar, where N is a finite set of nonterminal symbols; Σ is a finite set of terminal symbols; R is a finite set of productions R = {v : X → Y1Y2 . . . Yk} with head symbol X ∈ N and constituent symbols Yi ∈ N ∪ Σ for i = 1 . . . k and k > 0; S ∈ N is a distinguished start symbol, meaning that it does not appear on the right-hand side of any production; and w are the weights that parameterize the probability distribution defined by G. For a production v ∈ t in a parse tree t ∈ TG, we denote its region as Pv and its parent and children as pa(v) and ch(v), respectively, where TG is the set of possible parse trees under the grammar G. The labeling corresponding to the segmentation of the pixels in Pv for production v : X → Y1 . . . Yk is yv ∈ Y |Pv|v , where Yv = {Y1, . . . , Yk}. The region of any production v ∈ t is the set of pixels in Ppa(v) whose assigned label is the head of v, i.e., Pv = {p ∈ Ppa(v) : y pa(v) p = head(v)}, except for the production of the start symbol, which has
the entire image as its region. The probability of an image x is pw(x) = ∑
t∈TG pw(t,x), where the joint probability of parse tree t and the image is the product over all productions in t of the probability of choosing that production v and then segmenting its region Pv according to yv:
pw(t,x) = 1
Z exp(−Ew(t,x)) =
1 Z exp(− ∑ v∈t Evw(v,y v, head(v),Pv,x)).
Here, Z = ∑
t∈TG exp(−Ew(t,x)) is the partition function, w are the model parameters, and E is the energy function. In the following, we will simplify notation by omitting head(v), Pv , x, w, and superscript v from the energy function when they are clear from context. The energy of a production and its segmentation on the region Pv are given by a pairwise Markov random field (MRF) as E(v,yv) = ∑ p∈Pv θ v p(y v p ;w) + ∑ (p,q)∈Ev θ v pq(y v p , y v q ;w), where θ v p and θ v pq are the unary and pairwise costs of the segmentation MRF, {yvp : p ∈ Pv} is the labeling defining the segmentation of the pixels in the current region, and Ev are the edges in Pv . Without loss of generality we assume that Ev contains only one of (p, q) or (q, p), since the two terms can always be combined. Here, θvp is the per-pixel data cost and θvpq is the boundary term, which penalizes adjacent pixels within the same region that have different labels. We describe these terms in more detail below. In general, even computing the segmentation for a single production is intractable. In order to permit efficient inference, we require that θvpq satisfies the submodularity condition θ v pq(Y1, Y1) + θ v pq(Y2, Y2) ≤ θvpq(Y1, Y2) + θ v pq(Y2, Y1) for all productions v : X → Y1Y2 once the grammar has been converted to a grammar in which each production has only two constituents, which is always possible and in the worst case increases the grammar size quadratically (Jurafsky & Martin, 2000; Chomsky,
1959). We also require for every production v ∈ R and for every production c that is a descendant of v in the grammar that θvpq(y v p , y v q ) ≥ θcpq(ycp, ycq) for all possible labelings (yvp , yvq , ycp, ycq), where yvp , y v q ∈ Yv and ycp, ycq ∈ Yc. This condition ensures that segmentations for higher-level productions are submodular, no matter what occurs below them. It also encodes the reasonable assumption that higher-level abstractions are separated by stronger, shorter boundaries (relative to their size), while lower-level objects are more likely to be composed of smaller, more intricately-shaped regions.
The above model defines a sum-product network containing a sum node for each possible region of each nonterminal, a product node for each segmentation of each production of each possible region of each nonterminal, and a leaf function on the pixels of the image for each possible region of the image for each terminal symbol. The children of the sum node s for nonterminal Xs with region Ps are all product nodes r with a production vr : Xs → Y1 . . . Yk and region Pvr = Ps. Each product node corresponds to a labeling yvr of Pvr and the edge to its parent sum node has weight exp(−E(v,yvr ,Pvr )). The children of product node r are the sum or leaf nodes with matching regions that correspond to the constituent nonterminals or terminals of vr, respectively. Since the weights of the edges from a sum node to its children correspond to submodular energy functions, we call this a submodular sum-product network (SSPN).
A key benefit of SSPNs in comparison to previous grammar-based approaches is that regions can have arbitrary shapes and are not restricted to a small class of shapes such as rectangles (Poon & Domingos, 2011; Zhao & Zhu, 2011). This flexibility is important when parsing images, as realworld objects and abstractions can take any shape, but it comes with a combinatorial explosion of possible parses. However, by exploiting submodularity, we are able to develop an efficient inference algorithm for SSPNs, allowing us to efficiently parse images into a hierarchy of arbitrarily-shaped regions and objects, yielding a very expressive model class. This efficiency is despite the size of the underlying SSPN, which is in general far too large to explicitly instantiate.
2.1 MRF SEGMENTATION DETAILS
As discussed above, the energy of each segmentation of a region for a given production is defined by a submodular MRF E(v,yv) = ∑ p∈Pv θ v p(y v p ;w) + ∑ (p,q)∈Ev θ v pq(y v p , y v q ;w). The unary terms in E(v,yv) differ depending on whether the label yvp corresponds to a terminal or nonterminal symbol. For a terminal T ∈ Σ, the unary terms are a linear function of the image features θvp(yvp = T ;w) = wPCv +w > T φ U p , wherew PC v is an element of w that specifies the cost of v relative to other productions and φUp is a feature vector representing the local appearance of pixel p. In our experiments, φ U p is the output of a deep neural network. For labels corresponding to a nonterminal X ∈ N , the unary terms are θvp(y v p = X;w) = w PC v + θ c p(y c p), where c is the child production of v in the current parse tree that contains p, such that p ∈ Pc. This dependence makes inference challenging, because the choice of children in the parse tree itself depends on the region that is being parsed as X , which depends on the segmentation this unary is being used to compute.
The pairwise terms in E(v,yv) are a recursive version of the standard contrast-dependent pairwise boundary potential (e.g., Shotton et al. (2006)) defined for each production v and each pair of adjacent pixels p, q as θvpq(y v p , y v q ;w) = w BF v exp(−β−1||φBp −φBq ||2)·[yvp 6= yvq ]+θcpq(ycp, ycq;w), where β is half the average image contrast between all adjacent pixels in an image, wBFv is the boundary factor that controls the relative cost of this term for each production, φBp is the pairwise per-pixel feature vector, c is the same as in the unary term above, and [·] is the indicator function, which has value 1 when its argument is true and is 0 otherwise. For each pair of pixels (p, q), only one such term will ever be non-zero, because once two pixels are labeled differently at a node in the parse tree, they are placed in separate subtrees and thus never co-occur in any region below the current node. In our experiments, φBp are the intensity values for each pixel.
3 INFERENCE
Scene understanding (or semantic segmentation) requires labeling each pixel of an image with its semantic class. By constructing a grammar containing a set of nonterminals in one-to-one correspondence with the semantic labels and only allowing these symbols to produce terminals, we can recover the semantic segmentation of an image from a parse tree for this grammar. In the simplest case, a grammar need contain only one additional production from the start symbol to all other nonterminals. More generally, however, the grammar encodes rich structure about the relationships
- confusing part: not clear that X->Y->AB in subregion of LHS figure is just sub-selecting from existing parse of Y->AB over entire region - need to explain clearly what’s happening…
- confusing par : not clear that X->Y->AB in subregion of LHS figure is just sub-selecting from existing parse of Y->AB over entire region - eed to explain clearly what’s happening…
between image regions at various levels of abstraction, including concepts such as composition and subcategorization. Identifying the relevant structure and relationships for a particular image entails finding the best parse of an image x given a grammarG (or, equivalently, performing MAP inference in the corresponding SSPN), i.e., t∗ = arg maxt∈TG p(t|x) = arg mint∈TG ∑ v∈tE(v,y v,x).
In PCFGs over sentences (Jurafsky & Martin, 2000), the optimal parse can be recovered exactly in timeO(n3|G|) with the CYK algorithm (Hopcroft & Ullman, 1979), where n is the length of the sentence and |G| is the number of productions in the grammar, by iterating over all possible split points of the sentence and using dynamic programming to avoid recomputing sub-parses. Unfortunately, for images and other 2-D data types, there are 2n possible segmentations of the data for each binary production, rendering this approach infeasible in general. With an SSPN, however, it is possible to efficiently compute the approximate optimal parse of an image. In our algorithm, INFERSSPN, this is done by iteratively constructing parses of different regions in a bottom-up fashion.
3.1 PARSE TREE CONSTRUCTION
Given a production v : X → Y1Y2 and two parse trees t1, t2 over the same region P and with head symbols Y1, Y2, respectively, then for any labeling yv ∈ {Y1, Y2}|P| of P we can construct a third parse tree tX over region P with root production v, labeling yv , and subtrees t′1, t′2 over regions P1,P2, respectively, such that Pi = {p ∈ P : yvp = Yi} and t′i = ti ∩ Pi for each i, where the intersection of a parse tree and a region t ∩ P is the new parse tree resulting from intersecting P with the region at each node in t. Of course, the quality of the resulting parse tree, tX , depends on the particular labeling (segmentation) yv used. Recall that a parse tree t on region P has energy E(t,P) = ∑ v∈tE(v,y v,Pv), which can be written asE(t,P) = ∑ p∈P θ t p+ ∑ (p,q)∈E θ t pq , where
θtp = ∑ v∈t θ v p(y v p) · [p ∈ Pv] and θtpq = ∑ v∈t θ v pq(y v p , y v q ) · [(p, q) ∈ Ev]. This allows us to define the fusion operation, which is a key subroutine in INFERSSPN. Note that δij is the Kronecker delta.
Definition 1. For a production v : X → Y1, Y2 and two parse trees t1, t2 over region P with head symbols Y1, Y2 then tX is the fusion of t1 and t2 constructed from the minimum energy labeling yv = arg min
y∈Y|P|v E(v, t1, t2,y), where E(v, t1, t2,y) = ∑ p∈P θt1p · δypY1 + θt2p · δypY2 + ∑ (p,q)∈E θt1pq · δypY1 · δyqY1
+ θt2pq · δypY2 · δyqY2 + θvpq(Y1, Y2) · δypY1 · δyqY2 .
Figure 2a shows an example of fusing two parse trees to create a new parse tree. Although fusion requires finding the optimal labeling from an exponentially large set, the energy is submodular and can be efficiently optimized with a single graph cut. All proofs are presented in the appendix. Proposition 1. The energy E(v, t1, t2,yv) of the fusion of parse trees t1, t2 over region P with head symbols Y1, Y2 for a production v : X → Y1Y2 is submodular.
Once a parse tree has been constructed, INFERSSPN then improves that parse tree on subsequent iterations. The following result shows how INFERSSPN can improve a parse tree while ensuring that the energy of that parse tree never gets worse. Lemma 1. Given a labeling yv which fuses parse trees t1, t2 into t with root production v, energy E(t,P) = E(v, t1, t2,yv), and subtree regions P1 ∩ P2 = ∅ defined by yv , then any improvement
∆ in E(t1,P1) also improves E(t,P) by at least ∆, regardless of any change in E(t1,P\P1).
Finally, it will be useful to define the union t = t1 ∪ t2 of two parse trees t1, t2 that have the same production at their root but are over disjoint regions P1 ∩ P2 = ∅, as the parse tree t with region P = P1∪P2 and in which all nodes that co-occur in both t1 and t2 (i.e., have the same path to them from the root and have the same production) are merged to form a single node in t. In general, t may be an inconsistent parse tree, as the same symbol may be parsed as two separate productions, in which case we define the energy of the boundary terms between the pixels parsed as these separate productions to be infinite.
3.2 INFERSSPN
Pseudocode for our algorithm, INFERSSPN, is presented in Algorithm 1. INFERSSPN is an iterative bottom-up algorithm based on graph cuts (Kolmogorov & Zabih, 2004) that provably converges to a local minimum of the energy function. In its first iteration, INFERSSPN constructs a parse tree over the full image for each production in the grammar. The parse of each terminal production is trivial to construct and simply labels each pixel as the terminal symbol. The parse for every other production v : X → Y1Y2 is constructed by choosing productions for Y1 and Y2 and fusing their corresponding parse trees to get a parse of the image as X . Since the grammar is non-recursive, we can construct a directed acyclic graph (DAG) containing a node for each symbol and an edge from each symbol to each constituent of each production of that symbol and then traverse this graph from the leaves (terminals) to the root (start symbol), fusing the children of each production of each symbol when we visit that symbol’s node. Of course, to fuse parses of Y1 and Y2 into a parse of X , we need to choose which production of Y1 (and Y2) to fuse; this is done by simply choosing the production of Y1 (and Y2) that has the lowest energy over the current region. The best parse of the image, t̂, now corresponds to the lowest-energy parse of all productions of the start symbol.
Further iterations of INFERSSPN improve t̂ in a flexible manner that allows any of its productions or labelings to change, while also ensuring that its energy never increases. INFERSSPN does this by again computing parses of the full image for each production in the grammar. This time, however, when parsing a symbol X , INFERSSPN independently parses each region of the image that was parsed as any production of X in t̂ (none of these regions will overlap because the grammar is nonrecursive) and then parses the remainder of the image given these parses of subregions of the image, meaning that the pixels in these other subregions are instantiated in the MRF but fixed to the labels that the subregion parses specify. The parse of the image as X is then constructed as the union of these subregion parses. This procedure ensures that the energy will never increase (see Theorem 1 and Lemma 1), but also that any subtree of t̂ can be replaced with another subtree if it results in lower energy. Figure 2b shows a simple example of updating a parse of a region as X → Y Z. Further, this (re)parsing of subregions can again be achieved in a single bottom-up pass through the grammar DAG, resulting in a very efficient algorithm for SSPN inference. This is because each pixel only appears in at most one subregion for any symbol, and thus only ever needs to be parsed once per production. See Algorithm 1 for more details.
3.3 ANALYSIS
As shown in Theorem 1, INFERSSPN always converges to a local minimum of the energy function. Similar to other graph-cut-based algorithms, such as α-expansion (Boykov et al., 2001), INFERSSPN explores an exponentially large set of moves at each step, so the returned local minimum is much better than those returned by more local procedures, such as max-product belief propagation. Further, we observe convergence within a few iterations in all experiments, with the majority of the energy improvement occurring in the first iteration. Theorem 1. Given a parse (tree) t̂ of S over the entire image with energy E(t̂), each iteration of INFERSSPN constructs a parse (tree) t of S over the entire image with energy E(t) ≤ E(t̂) and since the minimum energy of an image parse is finite, INFERSSPN will always converge.
As shown in Proposition 2, each iteration of INFERSSPN takes time O(|G|c(n)), where n is the number of pixels in the image and c(n) is the complexity of the underlying graph cut algorithm used, which is low-order polynomial in the worst-case but nearly linear-time in practice (Boykov & Kolmogorov, 2004; Boykov et al., 2001). Proposition 2. Let c(n) be the time complexity of computing a graph cut on n pixels and |G| be the size of the grammar defining the SSPN, then each iteration of INFERSSPN takes time O(|G|c(n)).
Algorithm 1 Compute the (approximate) MAP assignment of the SSPN variables (i.e., the productions and labelings) defined by an image and a grammar. This is equivalent to parsing the image.
Input: The image x, a non-recursive grammar G = (N,Σ, R, S,w), and (optional) input parse t̂. Output: A parse of the image, t∗, with energy E(t∗,x) ≤ E(t̂,x).
1: function INFERSSPN(x, G, t̂) 2: T,E ← empty lists of parse trees and energies, respectively, both of length |R|+ |Σ| 3: for each terminal Y ∈ Σ do 4: T [Y ]← the trivial parse with all pixels parsed as Y 5: E[Y ]← ∑ p∈x w > Y φ U p
6: while the energy of any production of the start symbol S has not converged do 7: for each symbol X ∈ N , in reverse topological order do // as defined by the DAG of G 8: for each subtree t̂i of t̂ rooted at a production ui with head X do 9: Pi,yi ← the region that t̂i is over and its labeling in t̂i // {Pi} are all disjoint 10: for each production vj : X → Y1Y2 do // iterate over all productions of X 11: tij , eij ←FUSE(Pi,yi, vj , T ) // parse Pi as vj by fusing parses of Y1 and Y2 12: PX ← all pixels that are not in any region Pi 13: for each production vj : X → Y1Y2 do // iterate over all productions of X 14: yrand ← a random labeling of PX // use random for initialization 15: tX , eX ← FUSE(PX ,yrand, vj , T, (∪itij)) // parse PX as vj given (∪itij) 16: update lists: T [vj ]← (∪itij) ∪ tX and E[vj ]← ∑ i eij + eX for all vj with head X 17: t̂, ê← the production of S with the lowest energy in E and its energy 18: return t̂, ê
Input: A region P , a labeling y of P , a production v : X → Y1Y2, a list of parses T , and an optional parse tP of pixels not in P , used to set pairwise terms of edges that are leaving P . Output: A parse tree rooted at v over region P and the energy of that parse tree. 1: function FUSE(P,y, v, T, tP ) 2: for each Yi with i ∈ 1, 2 do 3: ui ← production of Yi in T with lowest energy over {p : yp = Yi} given tP 4: create submodular energy function E(v,y,P,x) on P from T [u1], T [u2], and tP 5: yv, ev ← (arg) miny E(v,y,P,x) // label each pixel in P as Y1 or Y2 using graph cuts 6: tv ← combine T [u1] and T [u2] according to yv and append v as the root 7: return tv, ev
Note that a straightforward application of α-expansion to image parsing that uses one label for every possible parse in the grammar requires an exponential number of labels in general.
INFERSSPN can be extended to productions with more than two constituents by simply replacing the internal graph cut used to fuse subtrees with a multi-label algorithm such as α-expansion. INFERSSPN would still converge because each subtree would still never decrease in energy. An algorithm such as QPBO (Kolmogorov & Rother, 2007) could also be used, which would allow the submodularity restriction to be relaxed. Finally, running INFERSSPN on the grammar containing k − 1 binary productions that results from converting a grammar with a single production on k > 2 constituents is equivalent to running α-expansion on the k constituents.
4 EXPERIMENTS
We evaluated INFERSSPN by parsing images from the Stanford background dataset (SBD) using grammars with generated structure and weights inferred from the pixel labels of the images we parsed. SBD is a standard semantic segmentation dataset containing images with an average size of 320 × 240 pixels and a total of 8 labels. The input features we used were from the Deeplab system (Chen et al., 2015; 2016) trained on the same images used for evaluation (note that we are not evaluating learning and thus use the same features for each algorithm and evaluate on the training data in order to separate inference performance from generalization performance). We compared INFERSSPN to α-expansion on a flat pairwise MRF and to max-product belief propagation (BP) on a multi-level (3-D) pairwise grid MRF. Details of these models are provided in the appendix. We note
that the flat encoding for α-expansion results in a label for each path in the grammar, where there are an exponential number of such paths in the height of the grammar. However, once α-expansion converges, its energy is within a constant factor of the global minimum energy (Boykov et al., 2001) and thus serves as a good surrogate for the true global minimum, which is intractable to compute.
We compared these algorithms by varying three different parameters: boundary strength (strength of pairwise terms), grammar height, and number of productions per nonterminal. Each grammar used for testing contained a start symbol, multiple layers of nonterminals, and a final layer of nonterminals in one-to-one correspondence with the eight terminal symbols, each of which had a single production that produces a region of pixels. The start symbol had one production for each pair of symbols in the layer below it, and the last nonterminal layer (ignoring the nonterminals for the labels) had productions for each pair of labels, distributed uniformly over this last nonterminal layer.
Boundary strength. Increasing the boundary strength of an MRF makes inference more challenging, as individual pixel labels cannot be easily flipped without large side effects. To test this, we constructed a grammar as above with 2 layers of nonterminals (not including the start symbol), each containing 3 nonterminal symbols with 4 binary productions to the next layer. We vary wBFv for all v and plot the mean average pixel accuracy returned by each algorithm (the x-axis is log-scale) in Figure 3a. INFERSSPN returns parses with almost identical accuracy (and energy) to α-expansion. BP also returns comparable accuracies, but almost always returns invalid parses with infinite energy (if it converges at all) that contain multiple productions of the same object or a production of some symbol Y even though a pixel is labeled as symbol X.
Grammar height. In general, the number of paths in the grammar is exponential in its height, so the height of the grammar controls the complexity of inference and thus the difficulty of parsing images. For this experiment, we set the boundary scale factor to 10 and constructed a grammar with four nonterminals per layer, each with three binary productions to the next layer. Figure 3b shows the effect of grammar height on total inference time (to convergence or a maximum number of iterations, whichever first occurred). As expected from Proposition 2, the time taken for INFERSSPN scales linearly with the height of the grammar, which is within a constant factor of the size of the grammar when all other parameters are fixed. Similarly, inference time for both α-expansion and BP scaled exponentially with the height of the grammar because the number of labels for both increases combinatorially. Again, the energies and corresponding accuracies achieved by INFERSSPN were nearly identical to those of α-expansion (see Figure 5 in the appendix). Productions per nonterminal. The number of paths in the grammar is also directly affected by the number of productions per symbol. For this experiment, we increased each pairwise term by a factor of 10 and constructed a grammar with 2 layers of nonterminals, each with 4 nonterminal symbols. Figure 3c shows the effect of increasing the number of productions per nonterminal, which again demonstrates that INFERSSPN is far more efficient than either α-expansion or BP as the complexity of the grammar increases, while still finding comparable solutions (see Figure 6 in the appendix).
5 CONCLUSION
This paper proposed submodular sum-product networks (SSPNs), a novel extension of sum-product networks that can be understood as an instantiation of an image grammar in which all possible parses of an image over arbitrary shapes are represented. Despite this complexity, we presented
INFERSSPN, a move-making algorithm that exploits submodularity in order to find the (approximate) MAP state of an SSPN, which is equivalent to finding the (approximate) optimal parse of an image. Analytically, we showed that INFERSSPN is both very efficient – each iteration takes time linear in the size of the grammar and the complexity of one graph cut – and convergent. Empirically, we showed that INFERSSPN achieves accuracies and energies comparable to α-expansion, which is guaranteed to return optima within a constant factor of the global optimum, while taking exponentially less time to do so.
We have begun work on learning the structure and parameters of SSPNs from data. This is a particularly promising avenue of research because many recent works have demonstrated that learning both the structure and parameters of sum-product networks from data is feasible and effective, despite the well-known difficulty of grammar induction. We also plan to apply SSPNs to additional domains, such as activity recognition, social network modeling, and probabilistic knowledge bases.
ACKNOWLEDGMENTS
AF would like to thank Robert Gens and Rahul Kidambi for useful discussions and insights, and Gena Barnabee for assisting with Figure 1 and for feedback on this document. This research was partly funded by ONR grant N00014-16-1-2697 and AFRL contract FA8750-13-2-0019. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ONR, AFRL, or the United States Government.
A PROOFS
Proposition 1. The energy E(v, t1, t2,yv) of the fusion of parse trees t1, t2 over region P with head symbols Y1, Y2 for a production v : X → Y1Y2 is submodular.
Proof. E(v, t1, t2) is submodular as long as 2 · θvpq(Y1, Y2) ≥ θt1pq + θt2pq , which is true by construction, since θvpq(y v p , y v q ) ≥ θcpq(ycp, ycq) for c any possible descendant of v and for all labelings.
Lemma 2. Given a labeling yv which fuses parse trees t1, t2 into t with root production v, energy E(t,P) = E(v, t1, t2,yv), and subtree regions P1 ∩ P2 = ∅ defined by yv , then any improvement ∆ in E(t1,P1) also improves E(t,P) by at least ∆, regardless of any change in E(t1,P\P1).
Proof. Since the optimal fusion can be found exactly, and the energy of the current labeling yv has improved by ∆, the optimal fusion will have improved by at least ∆.
Proposition 2. Let c(n) be the time complexity of computing a graph cut on n pixels and |G| be the size of the grammar defining the SSPN, then each iteration of INFERSSPN takes time O(|G|c(n)).
Proof. Let k be the number of productions per nonterminal symbol and N be the nonterminals. For each nonterminal, FUSE is called k times for each region and once for the remainder of the pixels. FUSE itself has complexity O(|P| + c(|P|) = O(c(|P|)) when called with region P . However, in INFERSSPN each pixel is processed only once for each symbol because no regions overlap, so the worst-case complexity occurs when each symbol has only one region, and thus the total complexity of each iteration of INFERSSPN is O(|N |k · c(n)) = O(|G|c(n)).
Theorem 2. Given a parse (tree) t̂ of S over the entire image with energy E(t̂), each iteration of INFERSSPN constructs a parse (tree) t of S over the entire image with energy E(t) ≤ E(t̂), and since the minimum energy of an image parse is finite, INFERSSPN will always converge.
Proof. We will prove by induction that for all nodes ni ∈ t̂ with corresponding subtree t̂i, region Pi, production vi : X → Y1Y2 and child subtrees t̂1, t̂2, that E(ti) ≤ E(t̂i) after one iteration for all ti = T [vi]∩Pi. Since this holds for every production of S over the image, this proves the claim. Base case. When t̂i is the subtree with region Pi and production vi : X → Y containing only a single terminal child, then by definition ti = T [vi]∩Pi = t̂i because terminal parses do not change given the same region. Thus, E(ti) = E(t̂i) and the claim holds. Induction step. Let vi : X → Y1Y2 be the production for a node in t̂i with subtrees t̂1, t̂2 over regions P1,P2, respectively, such that P1 ∪ P2 = Pi and P1 ∩ P2 = ∅, and suppose that for all productions u1j with head Y1 and all productions u2k with head Y2 and corresponding parse trees t1j = T [u1j ]∩P1 and t2k = T [u2k]∩P2, respectively, thatE(t1j) ≤ E(t̂1j) andE(t2k) ≤ E(t̂2k). Now, when FUSE is called on region P1 it will choose the subtrees t∗1j : j = arg minj E(t1j ,P1), and t∗2k : k = arg mink E(t2k,P2) and fuse these into t′i over P . However, from Lemma 1, we know that ti could at the very least simply reuse the labeling yv that partitions P into P1,P2 and in doing so return a tree t′i with energy E(t ′ i) ≤ E(t̂i), because each of its subtrees over their same regions has lower (or equal) energy to those in t̂. Finally, since t′i is computed independently of any other trees for region P and then placed into T [vi] as a union of other trees, then ti = T [vi]∩P = t′i, and the claim follows.
B ADDITIONAL EXPERIMENTAL RESULTS AND DETAILS
We compared INFERSSPN to running α-expansion on a flat pairwise MRF and to max-product belief propagation over a multi-level (3-D) pairwise grid MRF. Each label of the flat MRF corresponds to a possible path in the grammar from the start symbol to a production to one of its constituent symbols, etc, until reaching a terminal. In general, the number of such paths is exponential in the height of the grammar. The unary terms are the sum of unary terms along the path and the pairwise term for a pair of labels is the pairwise term of the first production at which their constituents differ. For any two labels with paths that choose a different production of the same symbol (and have the same path from the start symbol) we assign infinite cost to enforce the restriction that an object can only have a single production of it into constituents. Note that after convergence α-expansion is
guaranteed to be within a constant factor of the global minimum energy (Boykov et al., 2001) and thus serves as a good surrogate for the true global minimum, which is intractable to compute. The multi-layer MRF is constructed similarly. The number of levels in the MRF is equal to the height of the DAG corresponding to the grammar used. The labels at a particular level of the MRF are all (production, constituent) pairs that can occur at this height in the grammar. The pairwise term between the same pixel in two levels is 0 when the parent label’s constituent equals the child label’s production head, and∞ otherwise. Pairwise terms within a layer are defined as in the flat MRF with infinite cost for incompatible labels (i.e., two neighboring productions of the same symbol), unless two copies of that nonterminal could be produced at that level by the grammar.
All experiments were run on the same computer running an Intel Core i7-5960X with 8 cores and 128MB of RAM. Each algorithm was limited to a single thread.

The paper discusses sub modular sum-product networks as a tractable extension for classical sum-product networks. The proposed approach is evaluated on semantic segmentation tasks and some early promising results are provided.
Summary:
———
I think the paper presents a compelling technique for hierarchical reasoning in MRFs but the experimental results are not yet convincing. Moreover the writing is confusing at times. See below for details.
Quality: I think some of the techniques could be described more carefully to better convey the intuition.
Clarity: Some of the derivations and intuitions could be explained in more detail.
Originality: The suggested idea is great.
Significance: Since the experimental setup is somewhat limited according to my opinion, significance is hard to judge at this point in time.
Detailed comments:
———
1. I think the clarity of the paper would benefit significantly from fixes to inaccuracies. E.g., \alpha-expansion and belief propagation are not `scene-understanding algorithms’ but rather approaches for optimizing energy functions. Computing the MAP state of an SSPN in time sub-linear in the network size seems counterintuitive because it means we are not allowed to visit all the nodes in the network. The term `deep probabilistic model’ should probably be defined. The paper states that InferSSPN computes `the approximate MAP state of the SSPN (equivalently, the optimal parse of the image)’ and I’m wondering how the `approximate MAP state' can be optimal. Etc.
2. Albeit being formulated for scene understanding tasks, no experiments demonstrate the obtained results of the proposed technique. To assess the applicability of the proposed approach a more detailed analysis is required. More specifically, the technique is evaluated on a subset of images which makes comparison to any other approach impossible. According to my opinion, either a conclusive experimental evaluation using, e.g., IoU metric should be given in the paper, or a comparison to publicly available results is possible.
3. To simplify the understanding of the paper a more intuitive high-level description is desirable. Maybe the authors can even provide an intuitive visualization of their approach.


HUMAN PERCEPTION IN COMPUTER VISION / CONFERENCE SUBMISSIONS
Computer vision has made remarkable progress in recent years. Deep neural network (DNN) models optimized to identify objects in images exhibit unprecedented task-trained accuracy and, remarkably, some generalization ability: new visual problems can now be solved more easily based on previous learning. Biological vision (learned in life and through evolution) is also accurate and generalpurpose. Is it possible that these different learning regimes converge to similar problem-dependent optimal computations? We therefore asked whether the human system-level computation of visual perception has DNN correlates and considered several anecdotal test cases. We found that perceptual sensitivity to image changes has DNN mid-computation correlates, while sensitivity to segmentation, crowding and shape has DNN end-computation correlates. Our results quantify the applicability of using DNN computation to estimate perceptual loss, and are consistent with the fascinating theoretical view that properties of human perception are a consequence of architecture-independent visual learning. 1 QUICK EXPERT SUMMARY Considering the learned computation of ImageNet-trained DNNs, we find: • Large computation changes for perceptually salient image changes (Figure 1). • Gestalt: segmentation, crowding, and shape interactions in computation (Figure 2). • Contrast constancy: bandpass transduction in first layers is later corrected (Figure 3). These properties are reminiscent of human perception, perhaps because learned general-purpose classifiers (human and DNN) tend to converge.
1 QUICK EXPERT SUMMARY
Considering the learned computation of ImageNet-trained DNNs, we find:
• Large computation changes for perceptually salient image changes (Figure 1). • Gestalt: segmentation, crowding, and shape interactions in computation (Figure 2). • Contrast constancy: bandpass transduction in first layers is later corrected (Figure 3).
These properties are reminiscent of human perception, perhaps because learned general-purpose classifiers (human and DNN) tend to converge.
2 INTRODUCTION
Deep neural networks (DNNs) are a class of computer learning algorithms that have become widely used in recent years (LeCun et al., 2015). By training with millions of examples, such models achieve unparalleled degrees of task-trained accuracy (Krizhevsky et al., 2012). This is not unprecedented on its own - steady progress has been made in computer vision for decades, and to some degree current designs are just scaled versions of long-known principles (Lecun et al., 1998). In previous models, however, only the design is general-purpose, while learning is mostly specific to the context of a trained task. Interestingly, for current DNNs trained to solve a large-scale image recognition problem (Russakovsky et al., 2014), the learned computation is useful as a building block for drastically different and untrained visual problems (Huh et al., 2016; Yosinski et al., 2014).
For example, orientation- and frequency-selective features (Gabor patches) can be considered general-purpose visual computations. Such features are routinely discovered by DNNs (Krizhevsky et al., 2012; Zeiler & Fergus, 2013), by other learning algorithms (Hinton & Salakhutdinov, 2006;
∗https://sites.google.com/site/rondekelhomepage/
Lee et al., 2008; 2009; Olshausen & Field, 1997), and are extensively hard-coded in computer vision (Jain & Farrokhnia, 1991). Furthermore, a similar computation is believed to underlie the spatial response properties of visual neurons of diverse animal phyla (Carandini et al., 2005; DeAngelis et al., 1995; Hubel & Wiesel, 1968; Seelig & Jayaraman, 2013), and is evident in human visual perception (Campbell & Robson, 1968; Fogel & Sagi, 1989; Neri et al., 1999). This diversity culminates in satisfying theoretical arguments as to why Gabor-like features are so useful in general-purpose vision (Olshausen, 1996; Olshausen & Field, 1997).
As an extension, general-purpose computations are perhaps of universal use. For example, a dimensionality reduction transformation that optimally preserves recognition-relevant information may constitute an ideal computation for both DNN and animal. More formally, different learning algorithms with different physical implementations may converge to the same computation when similar (or sufficiently general) problems are solved near-optimally. Following this line of reasoning, DNN models with good general-purpose computations may be computationally similar to biological visual systems, even more so than less accurate and less general biologically plausible simulations (Kriegeskorte, 2015; Yamins & DiCarlo, 2016).
Related work seems to be consistent with computation convergence. First, different DNN training regimes seem to converge to a similar learned computation (Li et al., 2015; Zhou et al., 2014). Second, image representation may be similar in trained DNN and in biological visual systems. That is, when the same images are processed by DNN and by humans or monkeys, the final DNN computation stages are strong predictors of human fMRI and monkey electrophysiology data collected from visual areas V4 and IT (Cadieu et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014). Furthermore, more accurate DNN models exhibit stronger predictive power (Cadieu et al., 2014; Dubey & Agarwal, 2016; Yamins et al., 2014), and the final DNN computation stage is even a strong predictor of human-perceived shape discrimination (Kubilius et al., 2016). However, some caution is perhaps unavoidable, since measured similarity may be confounded with categorization consistency, view-invariance resilience, or similarity in the inherent difficulty of the tasks undergoing comparison. A complementary approach is to consider images that were produced by optimizing trained DNN-based perceptual metrics (Gatys et al., 2015a;b; Johnson et al., 2016; Ledig et al., 2016), which perhaps yields undeniable evidence of non-trivial computational similarity, although a more objective approach may be warranted.
Here, we quantify the similarity between human visual perception, as measured by psychophysical experiments, and individual computational stages (layers) in feed-forward DNNs trained on a large-scale image recognition problem (ImageNet LSVRC). Comparison is achieved by feeding the experimental image stimuli to the trained DNN and comparing a DNN metric (mean mutual information or mean absolute change) to perceptual data. The use of reduced (simplified and typically non-natural) stimuli ensures identical inherent task difficulty across compared categories and prevents confounding of categorization consistency with measured similarity. Perception, a systemlevel computation, may be influenced less by the architectural discrepancy (biology vs. DNN) than are neural recordings.
3 CORRELATE FOR IMAGE CHANGE SENSITIVITY
From a perceptual perspective, an image change of fixed size has different saliency depending on image context (Polat & Sagi, 1993). To investigate whether the computation in trained DNNs exhibits similar contextual modulation, we used the Local Image Masking Database (Alam et al., 2014), in which 1080 partially-overlapping images were subjected to different levels of the same random additive noise perturbation, and for each image, a psychophysical experiment determined the threshold noise level at which the added-noise image is discriminated from two noiseless copies at 75% (Figure 1a). Threshold is the objective function that is compared with an L1-distance correlate in the DNN representation. The scale of measured threshold was:
20 · log10 (
std (noise) T
) , (1)
where std (noise) is the standard deviation of the additive noise, and T is the mean image pixel value calculated over the region where the noise is added (i.e. image center).
The DNN correlate of perceptual threshold we used was the average L1 change in DNN computation between added-noise images and the original, noiseless image. Formally,
Li,n1 (I) = ∣∣∣ai (I + noise (n))− ai (I)∣∣∣ , (2)
where ai (X) is the activation value of neuron i during the DNN feedforward pass for input image X , and the inner average (denoted by bar) is taken over repetitions with random n-sized noise (noise is introduced at random phase spectra in a fixed image location, an augmentation that follows the between-image randomization described by Alam et al., 2014; the number of repetitions was 10 or more). Unless otherwise specified, the final L1 prediction is L i,n 1 averaged across noise levels (−40 to 25 dB with 5-dB intervals) and computational neurons (first within and then across computational stages). Using L1 averaged across noise levels as a correlate for the noise level of perceptual threshold is a simple approximation with minimal assumptions.
Results show that the L1 metric is correlated with the perceptual threshold for all tested DNN architectures (Figure 1b, 4a-c). In other words, higher values of the L1 metric (indicating larger changes in DNN computation due to image perturbation, consistent with higher perturbation saliency) are
correlated with lower values of measured perceptual threshold (indicating that weaker noise levels are detectable, i.e. higher saliency once more).
To quantify and compare predictive power, we considered the percent of linearly explained variability (R2). For all tested DNN architectures, the prediction explains about 60% of the perceptual variability (Tables 1, 2; baselines at Tables 3-5), where inter-person similarity representing theoretical maximum is 84% (Alam et al., 2014). The DNN prediction is far more accurate than a prediction based on simple image statistical properties (e.g. RMS contrast), and is on par with a detailed perceptual model that relies on dozens of psychophysically collected parameters (Alam et al., 2014). The Spearmann correlation coefficient is much higher compared with the perceptual model (with an absolute SROCC value of about 0.79 compared with 0.70, Table 1), suggesting that the L1 metric gets the order right but not the scale. We did not compare these results with models that fit the experimental data (e.g. Alam et al., 2015; Liu & Allebach, 2016), since the L1 metric has no explicit parameters. Also, different DNN architectures exhibited high similarity in their predictions (R2 of about 0.9, e.g. Figure 4d).
Prediction can also be made from isolated computational stages, instead of across all stages as before. This analysis shows that the predictive power peaks mid-computation across all tested image scales (Figure 1c). This peak is consistent with use of middle DNN layers to optimize perceptual metrics (Gatys et al., 2015a;b; Ledig et al., 2016), and is reminiscent of cases in which low- to mid-level vision is the performance limiting computation in the detection of at-threshold stimuli (Campbell & Robson, 1968; Del Cul et al., 2007).
Finally, considering the images for which the L1-based prediction has a high error suggests a factor which causes a systematic inconsistency with perception (Figures 1d, 6). This factor may be related to the mean image luminance: by introducing noise perturbations according to the scale of Equation 1, a fixed noise size (in dB) corresponds to smaller pixel changes in dark compared with bright images. (Using this scales reflects an assumption of multiplicative rather than additive conservation; this assumption may be justified for the representation at the final but perhaps not the intermediate computational stages considering the log-linear contrast response discussed in Section 5). Another factor may the degree to which image content is identifiable.
4 CORRELATE FOR MODULATION OF SENSITIVITY BY CONTEXT
The previous analysis suggested gross computational similarity between human perception and trained DNNs. Next, we aimed to extend the comparison to more interpretable properties of perception by considering more highly controlled designs. To this end, we considered cases in which a static background context modulates the difficulty of discriminating a foreground shape, despite no spatial overlap of foreground and background. This permits interpretation by considering the cause of the modulation.
We first consider segmentation, in which arrangement is better discriminated for arrays of consistently oriented lines compared with inconsistently oriented lines (Figure 2a) (Pinchuk-Yacobi et al., 2016). Crowding is considered next, where surround clutter that is similar to the discriminated target leads to deteriorated discrimination performance (Figure 2b) (Livne & Sagi, 2007). Last to be addressed is object superiority, in which a target line location is better discriminated when it is in a shape-forming layout (Figure 2c) (Weisstein & Harris, 1974). In this case, clutter is controlled by having the same fixed number of lines in context. To measure perceptual discrimination, these works introduced performance-limiting manipulations such as location jittering, brief presentation, and temporal masking. While different manipulations showed different measured values, order-of-difficulty was typically preserved. Here we changed all the original performance-limiting manipulations to location jittering (whole-shape or element-wise, see Section 8.4).
To quantify discrimination difficulty in DNNs, we measured the target-discriminative information of isolated neurons (where performance is limited by location jittering noise), then averaged across all neurons (first within and then across computational layer stages). Specifically, for each neuron, we measured the reduction in categorization uncertainty due to observation, termed mutual information (MI): MI (Ai;C) = H (C)−H (C|Ai) , (3) where H stands for entropy, and Ai is a random variable for the value of neuron i when the DNN processes a random image from a category defined by the random variable C. For example, if a neuron gives a value in the range of 100.0 to 200.0 when the DNN processes images from category A, and 300.0 to 400.0 for category B, then the category is always known by observing the value, and so mutual information is high (MI=1 bits). On the other extreme, if the neuron has no discriminative task information, then MI=0 bits. To measure MI, we quantized activations into eight equal-amount bins, and used 500 samples (repetitions having different location jittering noise) across categories. The motivation for this correlate is the assumption that the perceptual order-of-difficulty reflects the quantity of task-discriminative information in the representation.
Results show that, across hundreds of configurations (varying pattern element size, target location, jitter magnitude, and DNN architecture; see Section 8.4), the qualitative order of difficulty in terms of the DNN MI metric is consistent with the order of difficulty measured in human psychophysical experiments, for the conditions addressing segmentation and crowding (Figures 2d, 7; for baseline models see Figure 8). It is interesting to note that the increase in similarity develops gradually along different layer types in the DNN computation (i.e. not just pooling layers), and is accompanied by a gradual increase in the quantity of task-relevant information (Figure 2e-g). This indicates a link between task relevance and computational similarity for the tested conditions. Note that unlike the evident increase in isolated unit task information, the task information from all units combined decreases by definition along any computational hierarchy. An intuition for this result is that the total hidden information decreases, while more accessible per-unit information increases.
For shape formation, four out of six shapes consistently show order of difficulty like perception, and two shapes consistently do no (caricature at Figure 2h; actual data at Figure 9).
5 CORRELATE FOR CONTRAST SENSITIVITY
A cornerstone of biological vision research is the use of sine gratings at different frequencies, orientations, and contrasts (Campbell & Robson, 1968). Notable are results showing that the lowest perceivable contrast in human perception depends on frequency. Specifically, high spatial frequencies are attenuated by the optics of the eye, and low spatial frequencies are believed to be attenuated due to processing inefficiencies (Watson & Ahumada, 2008), so that the lowest perceivable contrast is found at intermediate frequencies. (To appreciate this yourself, examine Figure 3a). Thus, for low-contrast gratings, the physical quantity of contrast is not perceived correctly: it is not preserved across spatial frequencies. Interestingly, this is corrected for gratings of higher contrasts, for which perceived contrast is more constant across spatial frequencies (Georgeson & Sullivan, 1975).
The DNN correlate we considered is the mean absolute change in DNN representation between a gray image and sinusoidal gratings, at all combinations of spatial frequency and contrast. Formally, for neurons in a given layer, we measured:
L1(contrast, frequency) = 1
Nneurons Nneurons∑ i=1 ∣∣∣ai (contrast, frequency)− ai (0, 0)∣∣∣ , (4)
where ai (contrast, frequency) is the average activation value of neuron i to 250 sine images (random orientation, random phase), ai (0, 0) is the response to a blank (gray) image, and Nneurons is the number of neurons in the layer. This measure reflects the overall change in response vs. the gray image.
Results show a bandpass response for low-contrast gratings (blue lines strongly modulated by frequency, Figures 3, 10), and what appears to be a mostly constant response at high contrast for end-computation layers (red lines appear more invariant to frequency), in accordance with perception.
We next aimed to compare these results with perception. Data from human experiments is generally iso-output (i.e. for a pre-set output, such as 75% detection accuracy, the input is varied to find the value which produce the preset output). However, the DNN measurements here are iso-input (i.e. for a fixed input contrast the L1 is measured). As such, human data should be compared to the interpoalted inverse of DNN measurements. Specifically, for a set output value, the interpolated contrast value which produce the output is found for every frequency (Figure 11). This analysis permits quantifying the similarity of iso-output curves for human and DNN, measured here as the percent of log-Contrast variability in human measurements which is explained by the DNN predictions. This showed a high explained variability at the end computation stage (prob layer, R2 = 94%), but importantly, a similarly high value at the first computational stage (conv1 1 layer, R2 = 96%). Intiutively, while the ”internal representation” variability in terms of L1 is small, the iso-output number-of-input-contrast-cahnges variability is still high. For example. for the prob layer, about the same L1 is measured for (Contrast=1,freq=75) and for (Contrast=0.18,freq=12).
An interesting, unexpected observation is that the logarithmically spaced contrast inputs are linearly spaced at the end-computation layers. That is, the average change in DNN representation scales logarithmically with the size of input change. This can be quantified by the correlation of output L1 with log Contrast input, which showed R2 = 98% (averaged across spatial frequencies) for prob, while much lower values were observed for early and middle layers (up to layer fc7). The same computation when scrambling the learned parameters of the model showed R2 = 60%. Because the degree of log-linearity observed was extremely high, it may be an important emergent property of the learned DNN computation, which may deserve further investigation. However, this property is only reminiscent and not immediately consistent with the perceptual power-law scaling (Gottesman et al., 1981).
6 DISCUSSION
6.1 HUMAN PERCEPTION IN COMPUTER VISION
It may be tempting to believe that what we see is the result of a simple transformation of visual input. Centuries of psychophysics have, however, revealed complex properties in perception, by crafting stimuli that isolate different perceptual properties. In our study, we used the same stimuli to investigate the learned properties of deep neural networks (DNNs), which are the leading computer vision algorithms to date (LeCun et al., 2015).
The DNNs we used were trained in a supervised fashion to assign labels to input images. To some degree, this task resembles the simple verbal explanations given to children by their parents. Since human perception is obviously much richer than the simple external supervision provided, we were not surprised to find that the best correlate for perceptual saliency of image changes is a part of the DNN computation that is only supervised indirectly (i.e. the mid-computation stage). This similarity is so strong, that even with no fine-tuning to human perception, the DNN metric is competitively accurate, even compared with a direct model of perception.
This strong, quantifiable similarity to a gross aspect of perception may, however, reflect a mix of similarities and discrepancies in different perceptual properties. To address isolated perceptual effects, we considered experiments that manipulate a spatial interaction, where the difficulty of discriminating a foreground target is modulated by a background context. Results showed modulation of DNN target diagnostic, isolated unit information, consistent with the modulation found in perceptual discrimination. This was shown for contextual interactions reflecting grouping/segmentation (Harris et al., 2015), crowding/clutter (Livne & Sagi, 2007; Pelli et al., 2004), and shape superiority (Weisstein & Harris, 1974). DNN similarity to these groupings/gestalt phenomena appeared at the end-computation stages.
No less interesting, are the cases in which there is no similarity. For example, perceptual effects related to 3D (Erdogan & Jacobs, 2016) and symmetry (Pramod & Arun, 2016) do not appear to have a strong correlate in the DNN computation. Indeed, it may be interesting to investigate the influence of visual experience in these cases. And, equally important, similarity should be considered in terms of specific perceptual properties rather than as a general statement.
6.2 RECURRENT VS. FEEDFORWARD CONNECTIVITY
In the human hierarchy of visual processing areas, information is believed to be processed in a feedforward sweep, followed by recurrent processing loops (top-down and lateral) (Lamme & Roelfsema, 2000). Thus, for example, the early visual areas can perform deep computations. Since mapping from visual areas to DNN computational layers is not simple, it will not be considered here. (Note that ResNet connectivity is perhaps reminiscent of unrolled recurrent processing).
Interestingly, debate is ongoing about the degree to which visual perception is dependent on recurrent connectivity (Fabre-Thorpe et al., 1998; Hung et al., 2005): recurrent representations are obviously richer, but feedforward computations converge much faster. An implicit question here regarding the extent of feasible feed-forward representations is, perhaps: Can contour segmentation, contextual influences, and complex shapes be learned? Based on the results reported here for feedforward DNNs, a feedforward representation may seem sufficient. However, the extent to which this is true may be very limited. In this study we used small images with a small number of lines, while effects such as contour integration seem to take place even in very large configurations (Field et al., 1993). Such scaling seems more likely in a recurrent implementation. As such, a reasonable hypothesis may be that the full extent of contextual influence is only realizable with recurrence, while feedforward DNNs learn a limited version by converging towards a useful computation.
6.3 IMPLICATIONS AND FUTURE WORK
6.3.1 USE IN BRAIN MODELING
The use of DNNs in modeling of visual perception (or of biological visual systems in general) is subject to a tradeoff between accuracy and biological plausibility. In terms of architecture, other deep models better approximate our current understanding of the visual system (Riesenhuber &
Poggio, 1999; Serre, 2014). However, the computation in trained DNN models is quite generalpurpose (Huh et al., 2016; Yosinski et al., 2014) and offers unparalleled accuracy in recognition tasks (LeCun et al., 2015). Since visual computations are, to some degree, task- rather than architecturedependent, an accurate and general-purpose DNN model may better resemble biological processing than less accurate biologically plausible ones (Kriegeskorte, 2015; Yamins & DiCarlo, 2016). We support this view by considering a controlled condition in which similarity is not confounded with task difficulty or categorization consistency.
6.3.2 USE IN PSYCHOPHYSICS
Our results imply that trained DNN models have good predictive value for outcomes of psychophysical experiments, permitting a zero-cost first-order approximation. Note, however, that the scope of such simulations may be limited, since learning (Sagi, 2011) and adaptation (Webster, 2011) were not considered here.
Another fascinating option is the formation of hypotheses in terms of mathematically differentiable trained-DNN constraints, whereby it is possible to efficiently solve for the visual stimuli that optimally dissociate the hypotheses (see Gatys et al. 2015a;b; Mordvintsev et al. 2015 and note Goodfellow et al. 2014; Szegedy et al. 2013). The conclusions drawn from such stimuli can be independent of the theoretical assumptions about the generating process (for example, creating new visual illusions that can be seen regardless of how they were created).
6.3.3 USE IN ENGINEERING (A PERCEPTUAL LOSS METRIC)
As proposed previously (Dosovitskiy & Brox, 2016; Johnson et al., 2016; Ledig et al., 2016), the saliency of small image changes can be estimated as the representational distance in trained DNNs. Here, we quantified this approach by relying on data from a controlled psychophysical experiment (Alam et al., 2014). We found the metric to be far superior to simple image statistical properties, and on par with a detailed perceptual model (Alam et al., 2014). This metric can be useful in image compression, whereby optimizing degradation across image sub-patches by comparing perceptual loss may minimize visual artifacts and content loss.
ACKNOWLEDGMENTS
We thank Yoram Bonneh for his valuable questions which led to much of this work.
7 APPENDIX: FIGURES AND TABLES
8 APPENDIX: EXPERIMENTAL SETUP
8.1 DNN MODELS
To collect DNN computation snapshots, we used MATLAB with MatConvNet version 1.0-beta20 (Vedaldi & Lenc, 2015). All MATLAB code will be made available upon acceptance of this manuscript. The pre-trained DNN models we have used are: CaffeNet (which is a variant of AlexNet provided in Caffe, Jia et al., 2014), GoogLeNet (Szegedy et al., 2014), VGG-19 (Simonyan & Zisserman, 2014), and ResNet-152 (He et al., 2015). The models were trained on the same ImageNet LSVRC. The CaffeNet model was trained using Caffe with the default ImageNet training parameters (stopping at iteration 310, 000) and imported into MatConvNet. For the GoogLeNet model, we used the imported pre-trained reference-Caffe implementation. For VGG-19 and ResNet-152, we used the imported pre-trained original versions. In all experiments input image size was 224× 224 or 227× 227.
8.2 BASELINE MODELS
As baselines to compare with pre-trained DNN models, we consider: (a) a multiscale linear filter bank of Gabor functions, (b) a steerable-pyramid linear filter bank (Simoncelli & Freeman, 1995), (c) the VGG-19 model for which the learned parameters (weights) were randomly scrambled within layer, and (d) the CaffeNet model at multiple time points during training. For the Gabor decomposition, the following Gabor filters were used: all compositions of σ = {1, 2, 4, 8, 16, 32, 64}px, λ = {1, 2} · σ, orientation= {0, π/3, 2π/3, π, 4π/3, 5π/3}, and phase= {0, π/2}.
8.3 IMAGE PERTURBATION EXPERIMENT
The noiseless images were obtained from Alam et al. (2014). In main text, ”image scale” refers to percent coverage of DNN input. Since size of original images (149 × 149) is smaller than DNN input of (224× 224) or (227× 227), the images were resized by a factor of 1.5 so that 100% image scale covers approximately the entire DNN input area.
Human psychophysics and DNN experiments were done for nearly identical images. A slight discrepancy relates to how the image is blended with the background in the special case where the region where noise is added has no image surround at one or two side. In these sides (which depend on the technical procedure with which images were obtained, see Alam et al., 2014), the surround blending here was hard, while the original was smooth.
8.4 BACKGROUND CONTEXT EXPERIMENT
8.4.1 SEGMENTATION
The images used are based on the Texture Discrimination Task (Karni & Sagi, 1991). In the variant considered here (Pinchuk-Yacobi et al., 2015), subjects were presented with a grid of lines, all of which were horizontal, except two or three that were diagonal. Subjects discriminated whether the arrangement of diagonal lines is horizontal or vertical, and this discrimination was found to be more difficult when the central line is horizontal rather than diagonal (”Hard” vs. ”Easy” in Figure 2a). To limit human performance in this task, two manipulations were applied: (a) the location of each line in the pattern was jittered, and (b) a noise mask was presented briefly after the pattern. Here we only retained (a).
A total of 90 configurations were tested, obtained by combinations of the following alternatives:
• Three scales: line length of 9, 12.3, or 19.4 px (number of lines co-varied with line length, see Figure 12).
• Three levels of location jittering, defined as a multiple of line length: {1, 2, 3} · 0.0625 · l px, where l is the length of a line in the pattern. Jittering was applied separately to each line in the pattern.
• Ten locations of diagonal lines: center, random, four locations of half-distance from center to corners, four locations of half-distance from center to image borders.
For each configuration, the discriminated arrangement of diagonal lines was either horizontal or vertical, and the central line was either horizontal or diagonal (i.e. hard or easy).
8.4.2 CROWDING
The images used are motivated by the crowding effect (Livne & Sagi, 2007; Pelli et al., 2004).
A total of 90 configurations were tested, obtained by combinations of the following alternatives:
• Three scales: font size of 15.1, 20.6, or 32.4 px (see Figure 13). • Three levels of discriminated-letter location jittering, defined as a multiple of font size: {1, 2, 3} · 0.0625 · l px, where l is font size. The jitter of surround letters (M, N, S, and T) was fixed (i.e. the background was static).
• Ten locations: center, random, four locations of half-distance from center to corners, four locations of half-distance from center to image borders.
For each configuration, the discriminated letter was either A, B, C, D, E, or F, and the background was either blank (easy) or composed of the letters M, N, S, and T (hard).
8.4.3 SHAPE
The images used are based on the object superiority effect by Weisstein & Harris (1974), where discriminating a line location is easier when combined with surrounding lines a shape is formed.
A total of 90 configurations were tested, obtained by combinations of the following alternatives:
• Three scales: discriminated-line length of 9, 15.1, or 22.7 px (see Figure 14). • Five levels of whole-pattern location jittering, defined as a multiple of discriminated-line
length: {1, 2, 5, 10, 15} · 0.0625 · l px, where l is the length of the discriminated line.
• Six ”hard” background line layouts (patterns b-f of their Figure 2 and the additional pattern f of their Figure 3 in Weisstein & Harris, 1974). The ”easy” layout was always the same (pattern a).
For each configuration, the line whose location is discriminated had four possible locations (two locations are shown in Figure 2c), and the surrounding background line layout could compose a shape (easy) or not (hard).
8.5 CONTRAST SENSITIVITY EXPERIMENT
Used images depicted sine gratings at different contrast, spatial frequency, sine phase, and sine orientation combinations.

The author works to compare DNNs to human visual perception, both quantitatively and qualitatively.
Their first result involves performing a psychophysical experiment both on humans and on a model and then comparing the results (actually I think the psychophysical data was collected in a different work, and is just used here).   The specific psychophysical experiment determined, separately for each of a set of approx. 1110 images, what the noise level of additive noise would have to be to make a just-noticeable-difference for humans in discriminating the noiseless image from the noisy one.   The authors then define a metric on neural networks that allows them to measure what they posit might be a similar property for the networks.  They then correlate the pattern of noise levels between neural networks that the humans.    Deep neural networks end up being much better predictors of the human pattern of noise levels than simpler measure of image perturbation (e.g. RMS contrast).
A second result involves comparing DNNs to humans in terms of their pattern errors in a series of highly controlled experiments using stimuli that illustrate classic properties of human visual processing -- including segmentation, crowding and shape understanding.  They then used an information-theoretic single-neuron metric of discriminability to assess similar patterns of errors for the DNNs.   Again, top layers of DNNs were able to reproduce the human patterns of difficulty across stimuli, at least to some extent.
A third result involves comparing DNNs to humans in terms of their pattern of contrast sensitivity across a series of sine-grating images at different frequencies.  (There is a classic result from vision research as to what this pattern should be, so it makes a natural target for comparison to models.)   The authors define a DNN correlate for the propertie in terms of the cross-neuron average of the L1-distance between responses to a blank image and responses to a sinuisoid of each contrast and frequency.   They then qualitatively compare the results of this metric for DNNs models to known results from the literature on humans, finding that, like humans, there is an apparent bandpass response for low-contrast gratings and a mostly constant response at high contrast.
Pros:
    * The general concept of comparing deep nets to psychophysical results in a detailed, quantitative way, is really nice.
    * They nicely defined a set of "linking functions", e.g. metrics that express how a specific behavioral result is to be generated from the neural network.  (Ie. the L1 metrics in results 1 and 3 and the information-theoretic measure in result 2.)   The framework for setting up such linking functions seems like a great direction to me.
    * The actual psychophysical data seems to have been handled in a very careful and thoughtful way.   These folks clearly know what they're doing on the psychophysical end.
Cons:
    * To my mind, the biggest problem wit this paper is that that it doesn't say something that we didn't really know already.   Existing results have shown that DNNs are pretty good models of the human visual system in a whole bunch of ways, and this paper adds some more ways.    What would have been great would be:
         (a) showing that they metric of comparison to humans that was sufficiently sensitive that it could pull apart various DNN models, making one clearly better than the others.
         (b) identifying a wide gap between the DNNs and the humans that is still unfilled.   They sort of do this, since while the DNNs are good at reproducing the human judgements in Result 1, they are not perfect -- gap is between 60% explained variance and 84% inter-human consistency.    This 24% gap is potentially important, so I'd really like to see them have explored that gap more -- e.g. (i) widening the gap by identifying which images caused the gap most and focusing a test on those, or (ii) closing the gap by training a neural network to get the pattern 100% correct and seeing if that made better CNNs as measured on other metrics/tasks.
In other words, I would definitely have traded off not having results 2 and 3 for a deeper exploration of result 1.    I think their overall approach could be very fruitful, but it hasn't really been carried far enough here.
   * I found a few things confusing about the layout of the paper.  I especially found that the quantitative results for results 2 and 3 were not clearly displayed.   Why was figure 8 relegated to the appendix?  Where are the quantifications of model-human similarities for the data shown in Figure 8?  Isn't this the whole meat of their second result?   This should really be presented in a more clear way.
    * Where is the quantification of model-human similarity for the data show in Figure 3?  Isn't there a way to get the human contrast-sensitivity curve and then compare it to that of models in a more quantitively precise way, rather than just note a qualitative agreement?   It seems odd to me that this wasn't done.