Beyond Grids: Learning Graph Representations for Visual Recognition
We propose learning graph representations from 2D feature maps for visual recognition. Our method draws inspiration from region based recognition, and learns to transform a 2D image into a graph structure. The vertices of the graph define clusters of pixels ("regions"), and the edges measure the similarity between these clusters in a feature space. Our method further learns to propagate information across all vertices on the graph, and is able to project the learned graph representation back into 2D grids. Our graph representation facilitates reasoning beyond regular grids and can capture long range dependencies among regions. We demonstrate that our model can be trained from end-to-end, and is easily integrated into existing networks. Finally, we evaluate our method on three challenging recognition tasks: semantic segmentation, object detection and object instance segmentation. For all tasks, our method outperforms state-of-the-art methods.
The paper proposes to learn graph representations from visual data via graph convolutional unit (GCU). It transforms a 2D feature maps extracted from a neural network to a sample-dependent graph, where pixels with similar features form a vertex and edges measure affinity of vertices in a feature space. Then graph convolutions are applied to pass information along the edges of the graph and update the vertex features. Finally, the updated vertex features are projected back to 2D grids based on the pixel-to-vertex assignment. GCU can be integrated into existing networks allowing end-to-end training and capturing long-range dependencies among regions (vertices). The approach is evaluated on ADE20k for semantic segmentation task building upon PSPNet with ResNet-50/100 backbone and on MS COCO for object detection and instance segmentation tasks using Mask RCNN architecture. The method consistently improves over considered baselines (PSPNet, Mask RCNN) across different tasks.  Pros:  - A novel model has been presented for learning graph representations from 2D data. - The proposed method seems to be effective and efficient, allows end-to-end learning and can be integrated into existing networks.  Cons:  - The number of vertices has to be pre-specified. For efficiency reasons the number of vertices is kept small (set up to max 32 in the experiments). This results in very coarse, overly smooth outputs and missed finer details/objects (lines 240, 294), and will not be suitable for the applications that require pixel-accurate predictions. - No ablation study is provided. The paper lacks analysis of contributions of different steps to the performance, e.g. how much does initialization by clustering and regularization described in sec. 3.3. affect the final results? Also many design choices are not justified/analyzed in the paper, e.g. number of GCUs and number of vertices. Why were different number of vertices chosen for different task? How does the number of GCUs / number of vertices influence the results?  - The learnt groupings of pixels does not form meaningful semantic regions.  Minor comments/typos:  - Table 3,4 should be changed to Figure 3,4 - Line 293: mode -> modes  Overall, the paper presents a novel approach for learning graph representations from 2D data, which seems to be effective and flexible. However, it lacks the analysis and ablation study of different components. This work does not provide any insights on the design choices and the reader would have troubles transferring the proposed method to other tasks or new architectures. In addition, the proposed approach has a shortcoming that the number of vertices has to be pre-specified which negatively influences the quality of output predictions.