Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation
We propose a novel deep neural network architecture for semi-supervised semantic segmentation using heterogeneous annotations. Contrary to existing approaches posing semantic segmentation as a single task of region-based classification, our algorithm decouples classification and segmentation, and learns a separate network for each task. In this architecture, labels associated with an image are identified by classification network, and binary segmentation is subsequently performed for each identified label in segmentation network. The decoupled architecture enables us to learn classification and segmentation networks separately based on the training data with image-level and pixel-wise class labels, respectively. It facilitates to reduce search space for segmentation effectively by exploiting class-specific activation maps obtained from bridging layers. Our algorithm shows outstanding performance compared to other semi-supervised approaches with much less training images with strong annotations in PASCAL VOC dataset.
This paper describes a method of semantic segmentation, that can be applied particularly in cases where there are many image-level class labels but few segmented images.Classes are first identified using a classification network, then individually segmented using a second segmentation network.The second network is fed with the upper convolutional feature map of a the classification network, as well as the back-propagated activation of the individual class, which acts as a sort of seed.These are passed through a set of fc layers, which are reshaped and passed to a generating ``deconvolutional'' network that produces the segmentation for the class.Evaluations are performed using Pascal VOC.Overall, this pieces together some good components to create a nice segmentation system.However, some of the design choices could be further explored, and it would be nice to compare on more datasets than only Pascal VOC 2012.Questions follow:* I believe the same segmentation network is applied to all classes (i.e. weights are shared between classes), from l.219.But how much does this fact contribute to its performance when using few strongly annotated images, compared to the combinatorial data augmentation?* How many channels does the bridging layer output g_i^l have?Fig 2 suggests just 1.* While Fig 2 shows the bridging layer output is similar for each class, I wouldn't necessarily conclude that the "search space may not be as huge as our prejudice".Another possibility is that the class pattern is used in the first layer of the deconvolutional network to essentially perform the same unpooling (with switches) operation that forms the backpropagated map f^l_cls, so that the supplying the class provides more information than the spatial component (which is largely redundant).* Related, what happens if only f^l_cls is used (no f_spat), or if it is substituted with the class using a 1-hot encoding?What is the contribution of using this particular representation?* The text indicates the segmentation network is applied to each "identified label" -- how are these labels identified precisely from the classification net outputs?* A few additional references for semantic segmentation with convnets, which might also be included: - Farabet et al. "Learning Hierarchical Features for Scene Labeling" TPAMI 2013- Eigen & Fergus "Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture" arxiv 2014- Wang et al. "Deep Joint Task Learning for Generic Object Extraction" NIPS 2014