We present an image segmentation method that iteratively evolves a polygon. At each iteration, the vertices of the polygon are displaced based on the local value of a 2D shift map that is inferred from the input image via an encoder-decoder architecture. The main training loss that is used is the difference between the polygon shape and the ground truth segmentation mask. The network employs a neural renderer to create the polygon from its vertices, making the process fully differentiable. We demonstrate that our method outperforms the state of the art segmentation networks and deep active contour solutions in a variety of benchmarks, including medical imaging and aerial images.
EDIT: The rating changed from '1: Reject' to '6: Weak accept' after the rebuttal. See below for my reasoning.
The submission considers two-class image segmentation problems, where a closed-contour image region is to be specified as the 'object'/region of interest, vs. 'no-object'/background. The approach taken here is end-to-end learning with an active-contour type approach. The main loss, in contrast to other active contour approaches, contains a direct difference of the estimated polygon area vs. ground truth polygon area.
The applied method seems conceptually quite simple (as admitted by the authors in Section 5), and the neural rendering approach seems quite neat, but both method presentation (Section 3) and evaluation (Section 4) seem incomplete and leave significant open questions.
One of my main concerns is related to the fact that the displacement field is static and, according to Figure 1 and Algorithm 1, is evaluated only once per image.
If the displacement field J is not conditioned on the current polygon shape (and this does not seem to be the case), then I am wondering why T iterations in the sampling/rendering part are necessary at all. When only considering L_seg, the optimal solution should be found within one iteration, since the displacement field will be able to provide the optimal answer. So maybe these iterations are only necessary when L_B and L_K are incorporated?
In any case, it is unclear why even L_seg is accumulated (using unweighted mean) over all T iterations before being backpropagated. Does this mean that these iterations are not meant to yield shape improvements? Why is ||M^t-M|| not evaluated per iteration, for the purpose of minimization?
It is also not sufficiently clear whether M^t in Equation 4 is a filled polygon mask, or if the mask is just related to the boundary (with a certain width). In absence of explanatory image material, I am assuming the former.
Overall the method description remains weak, since obvious questions/concerns such as the above are not addressed.
The experimental results look good from a quantitative point of view, and indeed, the strongest baselines, e.g. DARNet, are outperformed significantly in many cases.
Section 4 mostly focuses on quantitative evaluation and lots of picture examples, but fails to give insight into particular behaviors, failure cases, etc.
The evaluation procedure is cast a bit into doubt by two things: 1) In Figure 4, the initializations (blue circles) between the DARNet method and the proposed method are very different in size. I am wondering if this then still constitutes a fair comparison, and I have some doubts there. 2) In Figure 6, the proposed method consistently looks much worse than the DARNet baseline (and, in contrast to the baseline, completely fails for 4 vertices), unless the colors were swapped in the description.
Overall, I do not think the submission is in a good enough shape for acceptance.
Minor remarks:
- The values for lambda_1 and lambda_2 seem to come out of thin air, and they also seem quite small. It needs to be mentioned how they were determined.
- Data augmentation by rotation seems to be missing several values (between 270 and 260 degrees) and also not evenly spaced. Is this a typo or on purpose? In the latter case, an explanation is needed, since this seems weird.
- Section 4.3: There is no "Figure 4.2", I assume you mean Figure 6, which otherwise remains unreferenced.
- Section 4.3, Ablation Study: Don't use the word "derivatives" when you're talking about variations.
- Section 4.3, Ablation Study: "even without no auxiliary loss" -> remove "no" or change "without" -> "with"
-------------
Post-rebuttal comments:
I have read the revised version, as well as the other reviews and all authors' comments. The inclusion of an evaluation on a larger-size data set is highly appreciated, and seems to indeed validate the robustness of the method. Typos were fixed, including the switched color descriptions in Figure 7 (which should not have passed initial submission in the first place, if the text had been proofread properly).
Several of the open questions (e.g. "Why is L_seg accumulated before backpropagation?", "Why is the algorithm iterative if the displacement map is computed only once, if not for the other loss terms?", "Choice of values for lambda_1, lambda_2", Initial diameter of initialization") have been somewhat addressed by the authors in the rebuttal comment, though not in great detail.
Based on the quality of the results across data sets, and because I believe that the timely publication of this rather simple method can benefit further research in this area, I have adjusted my score to a 'Weak accept'. That said, I still do not think it is a good manuscript, and my score should be seen as a massive benefit of the doubt toward the authors.
Most importantly, above questions have NOT been adequately addressed in the actual revised text. The authors claim they have "improved the manuscript considerably", but yet I see more reasoning for certain choices described in the comment here than in the actual manuscript. Most of the changes are in Section 2 and the new Section 4.3, but not much relevant to my comments changed in Section 3.
For example, balloon and curvature losses aside, it is still not clear why an iterative approach would be helpful past the first iteration. An ideal displacement map that is not conditioned on the polygon should point, for each pixel, straight to the closest contour pixel. It is clear to me that this may not be what is being learned when multiple iterations are forced, yet it is not addressed why multiple iterations should be beneficial. (I could see why they could be beneficial if the approach was conditioned on the polygon vertices, to avoid vertex collapsing, but it's not.)
A good submission preempts these kinds of questions by addressing them carefully. What seems crystal clear to the authors will not be crystal clear to every reader. The authors should be more careful to include their reasoning in the actual text, which I believe this is essential for proper, easy understanding of the paper.