Symbolic Graph Reasoning Meets Convolutions
Beyond local convolution networks, we explore how to harness various external human knowledge for endowing the networks with the capability of semantic global reasoning. Rather than using separate graphical models (e.g. CRF) or constraints for modeling broader dependencies, we propose a new Symbolic Graph Reasoning (SGR) layer, which performs reasoning over a group of symbolic nodes whose outputs explicitly represent different properties of each semantic in a prior knowledge graph. To cooperate with local convolutions, each SGR is constituted by three modules: a) a primal local-to-semantic voting module where the features of all symbolic nodes are generated by voting from local representations; b) a graph reasoning module propagates information over knowledge graph to achieve global semantic coherency; c) a dual semantic-to-local mapping module learns new associations of the evolved symbolic nodes with local representations, and accordingly enhances local features. The SGR layer can be injected between any convolution layers and instantiated with distinct prior graphs. Extensive experiments show incorporating SGR significantly improves plain ConvNets on three semantic segmentation tasks and one image classification task. More analyses show the SGR layer learns shared symbolic representations for domains/datasets with the different label set given a universal knowledge graph, demonstrating its superior generalization capability.
This paper is about a Symbolic Graph Reasoning (SGR) layer which can be injected between convolutional layers. The SGR layer employs a knowledge graph that allows the neural network to leverage semantic constraints derived from human knowledge.   The SGR is structured in 3 modules: 1) local-to-semantic voting module which maps local features to semantic nodes (i.e. the nodes of the knowledge graph) and outputs a D^c dimensional feature vector for each semantic node in the graph, 2) a graph reasoning module in which the semantic node features are concatenated with word-vector  features (derived from linguistic embeddings) projected to lower dimensionality linear layer. Finally, each semantic node representation is replaced by the average of the node representations of its neighbors (using the dot product with the row-normalized adjacency matrix of the knowledge graph). 3) the semantic-to-local mapping module reconstructs the local features starting from the  global representation of the semantic nodes in produced in the previous module.  The paper include experiments with the SGR module on the semantic segmentation datasets such as Coco-Stuff, ADE20K and PASCAL-Context and on the classification dataset CIFAR-100. The experimental evaluation is quite convincing and I find impressive that the performance of SGR/SGR 2-layer can get very close to the one of DenseNet using 7.5M/8.1M of parameters instead of 25.6M (see Table 5).  With respect to the novelty, the closest work is the one on Dynamic-structured semantic propagation networks [27] The author claim that "Our work takes an important next step beyond prior approaches in that it directly incorporates the reasoning over external knowledge graph into local feature learning, called as Symbolic Graph Reasoning (SGR) layer." However, also [27] can incorporate a graph and yields results which are a bit lower on Coco-stuff.  Strengths: + The paper is well written and easy to read. + The performance of the method is impressive and does not seem to be tailored to the specific tasks addressed.  Weaknesses: - It not clear what is novel compared to [27], which also incorporate a graph and yields similar results which are a bit lower on Coco-stuff  Questions: Q1)  Since [27] is missing in Table 2. Is there a reason because [27] can not be run on Pascal-Context? Q2) In [27] we can see experiments on the Cityscape and the Mapillary datasets would SGR work on those datasets? Q3) "The very recent DSSPN [27] directly designs a network layer for each parent concept. However, this method is hard to scale up for large-scale concept set and results in redundant predictions for pixels that unlikely belongs to a specific concept." Q3.1 Could you provide evidence of a large-scale concept set in which your method surpasses [27]? Q3.2 Is there a relationship between the size of the concept set and the number object classes that we want to segment? Q4) Could you clarify what is meant with the term "reasoning" in SGR? In particular, could we interpret the dot product of the row-normalized adjacency matrix with the semantic node features as a forward chaining step in an inference engine?   Minor Issues: - "people wear a hat and play guitar not vice-versa;" why not? - "orange is yellow color." orange is orange not yellow, unless I am missing something. - "Comparison with the state-of-the-arts" -> "Comparison with the state of the art"  —————— My concerns are all well addressed in the rebuttal. My opinion of the paper is increased.