It is usually hard for a learning system to predict correctly on the rare events, and there is no exception for segmentation algorithms. Therefore, we hope to build an alarm system to set off alarms when the segmentation result is possibly unsatisfactory. One plausible solution is to project the segmentation results into a low dimensional feature space, and then learn classifiers/regressors in the feature space to predict the qualities of segmentation results. In this paper, we form the feature space using shape feature which is a strong prior information shared among different data, so it is capable to predict the qualities of segmentation results given different segmentation algorithms on different datasets. The shape feature of a segmentation result is captured using the value of loss function when the segmentation result is tested using a Variational Auto-Encoder(VAE). The VAE is trained using only the ground truth masks, therefore the bad segmentation results with bad shapes become the rare events for VAE and will result in large loss value. By utilizing this fact, the VAE is able to detect all kinds of shapes that are out of the distribution of normal shapes in ground truth (GT). Finally, we learn the representation in the one-dimensional feature space to predict the qualities of segmentation results. We evaluate our alarm system on several recent segmentation algorithms for the medical segmentation task. The segmentation algorithms perform differently on different datasets, but our system consistently provides reliable prediction on the qualities of segmentation results.
This paper presents an approach to evaluate the quality of segmentations. To achieve this, a variational auto-encoder (VAE) is trained on the ground truth masks to extract shape-relevant information in the feature space, assuming that incorrect segmentations will be far from the normal distribution. Then, a regression model is trained to predict the quality of the segmentation based on the shape-learned features. The authors use several datasets focusing on pancreas segmentation to evaluate their quality-assessment approach, showing competitive performance with respect to other approaches.
The paper is well written, easy to follow in general, and the methodology is sound. Nevertheless, I have some concerns related to the applicability of this approach.
- Closely related works in the literature are missing:
There is a closely related recent work that used auto-encoders on the sets of ground-truth masks to build representations of shape and constrain the outputs of deep networks: Otkay et al., Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation, IEEE TMI 2017
This work does not focus directly on quality assessment. However, I believe the loss in this work, which evaluates the difference between the obtained segmentation (characterized by the outputs of a deep network) and an auto-encoder description of shape, can be used directly as a criterion for evaluating the quality of segmentation (on validation data) in term of consistency with the shape prior. I think this work is very closely related and should be discussed. 
Also, a quick google search provided some missing references related to this work. I think including comparisons to the recent work in [1], for example, would be appropriate. As the focus is on quality assessment of medical image segmentation, I would suggest a deeper review of the literature.
[1] Vanya V Valindria, Ioannis Lavdas, Wenjia Bai, Konstantinos Kamnitsas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Reverse classification accuracy: Predict- ing segmentation performance in the absence of ground truth. IEEE Transactions on Medical Imaging, 2017. 
[2] S. Chabrier, B. Emile, C. Rosenberger, and H. Laurent, “Unsupervised performance evaluation of image segmentation,” EURASIP Journal on Applied Signal Processing, vol. 2006, pp. 217–217, 2006. 
[3] Gao H, Tang Y, Jing L, Li H, Ding H. A Novel Unsupervised Segmentation Quality Evaluation Method for Remote Sensing Images. Sensors. 2017 Oct 24;17(10):2427.
- The proposed quality assessment uses the learned shape features.  Even though it is strong prior information, there  might be situations where the predicted segmentation might be plausible in terms of shape, but is not a good segmentation. 
-  I wonder how this approach works in problems with a high size/shape variation. For example, in the case of tumors, where their shape is unpredictable and each unknown case can be seen as a ‘rare’ example.
- To better capture the shape in the proposed approach, images need to be aligned, which limits the applicability of this approach to aligned volumes only. 
- This approach gives a global hint about a given segmentation result, as a whole. I think it would be more interesting to provide local information on a segmentation, as it may happen that a predicted contour is generally correct, but there are some crispy borders in some points due to low contrast, for example. Even though the quality assessment would say that the prediction is correct, the contour may be unusable for certain applications, where a minimal surface distance is required (e.g., radiotherapy).
- As the quality assessment is based on shape and not in image information, it would be interesting to see how accurately it predicts the performance on different image modalities (for example, the method is trained on ground truth masks corresponding to CT images and quality is assessed in segmentations performed in MRI).
If I understood correctly, comparison with other methods is done with the same dataset under the same conditions (i.e., all the images are pre-aligned). As the other methods might not have the limitation of requiring aligned images, it would be interesting to compare also the performances in this situation.
How the training (or the VAE) is adapted for DeepLab-3, as it is based on 2D convolutions?
Minor: The paper needs a proof-read to fix some issues (e.g. ‘the properties of F is encoded’)