We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.
Summary:
- key problem: neural architecture search (NAS) to improve both accuracy and runtime efficiency of deep nets for semantic segmentation;
- contributions: 1) a novel NAS search space leveraging multi-resolution branches, efficient operators ("zoomed convolutions"), and parametrized expansion ratios, 2) a decomposition and normalization of the latency objective to avoid a bias towards very fast but weak architectures, 3) a natural extension of the optimization problem to simultaneously search for teacher and student architectures in one pass, 4) a novel state-of-the-art efficient architecture (FasterSeg) found by the aforementioned NAS algorithm, 5) a detailed experimental evaluation on 3 datasets and an ablative analysis quantifying the benefits of the aforementioned contributions.
Recommendation: Accept.
Key reason 1: solid experimental results backing the claims.
- When compared to related efficient architectures, the proposed method results in competitive accuracy at significantly higher frame rates.
- This is validated on Cityscapes, CamVid, and BDD with the architecture found on Cityscapes. 
- The resulting architecture (FasterSeg) is actually interpretable and makes sense, extending the handcrafted architectures used as inspiration.
- The ablative analysis shows that the numerous individual contributions are significant, esp. the multi-branch formulation and student co-searching.
Key reason 2: well-motivated method with a collection of multiple novel contributions that are interesting and practical.
- The multi-resolution branches formulation is simple and extends typical NAS focusing on single paths through the supernet.
- Teacher/student co-searching via learning two sets of architectures in one supernet seems novel, simple, and effective. Always picking the largest expansion ratios for the teacher and applying a distillation loss in addition to the latency loss for the student is sensible and seems to beat the standard pruning approach at no significant extra cost during NAS.
- The zoomed convolution operator seems like a novel efficient alternative to (expensive) dilated convolutions. Although it is very simple (bilinear downsampling -> 3x3 conv -> bilinear upsampling), it is not commonly used as an operator (as far as I know), and yet is found to be a key part of the final architecture (Table 7 appendix I) due to its low latency. The closest related operator / block I could think of might be blocks found in stacked hourglass networks (Newell et al).
- The optimization of the expansion ratios using the Gumbel-Softmax trick is interesting, although this is also explored in the very recent paper by Shaw et al. 2019 (possibly the closest related work that should be discussed in a bit more depth in Section 2);
- Decomposing and normalizing the latency objective to avoid "architecture collapse" (convergence to anemic architectures stemming from certain architectural factors dominating latency) is principled and effective.
- Caveat regarding novelty: I could not find the ideas proposed here in the literature, but its hard to be sure due to 1) the recent explosion of NAS papers, 2) the simplicity of certain ideas (e.g., "zoomed convolutions").
Additional Feedback:
- how is the student trained after NAS? Is the teacher first retrained from scratch? Is the student retrained from scratch on the teacher (after NAS or retraining)? in general, more details on what happens after co-searching would be helpful;
- "human designed CNN architectures achieve superior accuracy performance nowadays": this is a surprising statement considering the cited NAS papers report performance improvements (e.g., Zoph and Le 2016);
- missing reference also using multi-scale NAS for efficient and accurate semantic segmentation: "Searching for Efficient Multi-Scale Architectures for Dense Image Prediction", Chen et al, NeurIPS 2018;
- missing reference on NAS for efficient semantic segmentation that also uses distillation: "Fast neural architecture search of compact semantic segmentation models via auxiliary cells", Nekrasov et al, CVPR 2019;
- missing reference on joint NAS and quantization: "Joint Neural Architecture Search and Quantization", Chen et al, arxiv 2018;
- "we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b)), i.e., convolutional layers are sequentially stacked in our network": "stacked" is confusing here;
- "we allow each cell to be individually searchable across the whole search space": what do you mean? Anything beyond each cell containing different operators after learning?
- if \alpha = \beta in eq. 6 of appendix C, then w and hence Target(m) does not depend on latency, isn't this a typo?
- "Gumbel-Max" is typically called "Gumbel-Softmax" (cf. "Categorical Reparameterization with Gumbel-Softmax", Jang et al, ICLR'17);
- typos: "find them contribute", "the closet competitor", "is popular dense predictions".