Overfitting is a common issue in machine learning, which can arise when the model learns to predict class membership using convenient but spuriouslycorrelated image features instead of the true image features that denote a class. These are typically visualized using saliency maps. In some object classification tasks such as for medical images, one may have some images with masks, indicating a region of interest, i.e., which part of the image contains the most relevant information for the classification. We describe a simple method for taking advantage of such auxiliary labels, by training networks to ignore the distracting features which may be extracted outside of the region of interest, on the training images for which such masks are available. This mask information is only used during training and has an impact on generalization accuracy in a dataset-dependent way. We observe an underwhelming relationship between controlling saliency maps and improving generalization performance.
Summary
---
(motivation)
CNN image classifiers tend to overfit to distractor patterns.
Perhaps these patterns are spatially local, such that in most images signal is at one
location while the noise models tend to overfit to is somewhere else.
If so, then generalization should improve if models are given additional
supervision (i.e., a mask identifying salient regions) that specifies where the signal is and is not.
This paper designs the Actdiff loss to realize this intuition.
(approach)
Actdiff:
1) The Actdiff loss requires a mask that highlights areas of the input
image which have signal and not distractor regions. It extracts features from
the original input image and its masked version then encourages the two features
to be similar at every layer of the CNN using an L2 loss.
Actdiff is compared to 5 other methods including a reconstruction loss and Gradmask (previous work).
(evaluation - synthetic dataset)
A synthetic dataset is constructed for a simple binary classification task based on the presence of simple shapes.
Two patterns can predict the correct class at train time, but one of the patterns is removed at test time so additional information (masks in this case)
is required to specify which pattern the classifier should use.
All the losses except Actdiff achieve at best 50% accuracy on the val set, but Actdiff gets 80% or more accuracy on 3 of 4 tested model variations.
This shows that Actdiff can effectively introduce the relevant masking information.
(evaluation - Medical Segmentation Decathalon)
This dataset provides 3 segmentation tasks (Liver, Cardiac, Pancreas), including ground truth masks for those images.
All 6 methods outperform all the others at least some of the time.
The conclusion is that adding mask information using Actdiff doesn't improve segmentation performance.
(evaluation - Multi-Site dataset)
A final task tries to construct another synthetic dataset out of real X-ray images
collected at two different places.
The train set is largely from one place and the test set is mostly from the other place, and masks are constructed so Actdiff can try to eliminate this bias.
Actdiff causes a negligible increase in performance.
The paper concludes that signals CNNs tend to fit to in this type of data are not very spatially distinct.
Strengths
---
There is some novelty in the approach. The actdiff loss makes sense and masking + activation mapping have not been tried together before to my knowledge.
Experiments follow a logical progression, starting by verifying the idea on a synthetic dataset, then moving to real data, and then evaluating on a half-synthetic dataset designed to debug the approach.
Experiments average over many random initializations.
The paper embraces its negative result.
Weaknesses
---
The approach is not very compelling to me:
* Implicit in this paper is that any information outside the mask is a distractor and any information inside is not a distractor. Why should the particular masks chosen for the experiments have this property? How can an expert know which features a model will find useful?
The paper's novelty is somewhat limited. The idea of regularizing using saliency maps has been explored and even applied to medical data like the MSD used here in Gradmask (one of the strong baselines this paper compares to). Activation matching is also common (e.g. [1]), though it has not been combined with masking before.
While the weights applied to the various losses are provided, it's not clear how they were tuned. In this case there may be lots of competing losses, so it's important to tune the weights somehow to ensure the tradeoff between losses is optimal.
The results (and the conclusions) suggest Actdiff is not very effective at increasing generalization. Table 2 reports test results on all datasets. In that table, each loss outperforms all the other losses in at least two cases (a case is a model-dataset pair).
[1]: Gatys, Leon A. et al. “A Neural Algorithm of Artistic Style.” ArXiv abs/1508.06576 (2015): n. pag.
Missing experiments:
* In the Multi-Site experiment, compare to what happens when the circular mask is applied to all images and not just those from one site. This is a necessary control to be sure that any benefits from masking are due to domain transfer and not other regularization effects. Either conclusion could be useful, but it would be nice to know.
Presentation weaknesses:
* In the synthetic dataset the model cannot tell the difference between correct and incorrect signals at train time. Therefore, I think there's no way for some of the baselines (plain classifier, autoencoder) to generalize correctly. Is that right? If so, it should be clear that comparisons to these baselines are not fair when discussing the synthetic evaluation in section 4.
Missing details / points of clarification:
* What is the Conv AE? I assume it is a CNN based autoencoder of some sort. A detailed description of the non-standard architectures would be useful for reproducibility, though probably only in the appendix.
* What are the lambda hyperparameters? I assume these are weights on the corresponding loss terms, but this is never made explicit.
* How does f(.) relate to the function o_l(.)? Is o_l(.) an intermediate step in f(.)?
* The MSD dataset is not clearly described. Is this a classification dataset where classes are different diseases? What do the ground truth masks capture?
* I think only Conv AE and UNet contain reconstruction losses. This presentation is a bit confusing since reconstruction loss was presented as another loss and it shows up in the tables implicitly based on the architecture being compared.
Suggestions
---
* This paper would be a bit more convincing if it started with a concrete example of the problem illustrated on some dataset (e.g., maybe an example from Gradmask). That may also help drive intuitions later on in the paper.
Preliminary Evaluation
---
Clarity: The paper is fairly clear.
Quality: Quality is mixed. Lots of relevant experiments are reported but they don't support clear conclusions and I'm not sure how well the models were tuned.
Originality: There is some novelty, but it is limited as discussed above.
Significance: I see limited significance.
For me this paper requires special scruitiny because it presents a negative result. Here are some factors that come to mind when thinking about whether to publish a negative result:
* Is the approach compelling? - This approach is not very compelling (e.g., comments about limited novelty and lack of concrete examples to boost intuition).
* Are the experiments thorough? - The experiments could be significantly more thorough (e.g., comments about tuning lambda).
* Will readers learn something useful? - This paper may help researchers trying to leverage similar intuitions, but it won't be very useful outside this audience.
* Does the paper present experiments that promote deeper understanding of why the approach failed? - This paper makes significant reasonable steps in that direction with sections 4 and 6, but I was still a bit dissapointed with the conclusions of these sections.
* Does the paper discuss alternative approaches that were investigated? - Many alterantive approaches were considered and their performance reported.
Overall I think this paper is close but fails to meet the bar because it does a bit worse than expected on most criteria above.