Dynamic Scale Inference by Entropy Minimization
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.
Summary: The following work proposes a test-time optimization over scales to improve semantic segmentation. Specifically, at test time, they iteratively optimize over the score and scale parameters of Shellhamer et al 2019, where a Gaussian receptive field is used to allow for dynamic scale adaptation of each convolutional layer. They optimize the parameters with respect to an entropy minimization objective. Experiments on PASCAL VOC, reported at multiple scales, demonstrate improvements in IOU over the vanilla architecture. Strengths: -The work was well-motivated -Formulation is pretty elegant and outperforms the baseline Weaknesses: While I liked the somewhat elegant formulation of dynamic test-time scaling proposed by the following work, I don*t think this work introduced many novel results nor insights -Multiscale test-time inference is already standard in state-of-the-art architectures such as DeepLab. Specifically, DeepLab runs inference at multiple scales, then max pools the logit responses across all scales. (see Table 3 of https://arxiv.org/pdf/1511.03339.pdf) -Using entropy as a measure of network uncertainty is a good idea, but also not a novel finding. -COCO and Cityscapes would probably have been better choices for datasets with larger variations in scale Improvements: -Try comparing against the multiscale logit-max-pooling inference procedure as a baseline -- or demonstrate that the proposed technique can further improve upon the results of the logit-pooling technique. -Some details weren*t very well explained. Specifically, what is the task parameter for? ** Post Rebuttal Response I*d like to thank the authors for clarifying some points in their response. Overall, I maintain that I think the optimization-based scale inference solution they present is interesting from an implementation standpoint, but the findings in this work did not yield sufficient new insight for the task. While I agree that their approach is fairly different from common approaches such as discrete image pyramids, a thorough quantitative comparison of these differences would make this work a lot stronger. As such, I maintain my original rating.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Dynamic Scale Inference by Entropy Minimization
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: The following work proposes a test-time optimization over scales to improve semantic segmentation. Specifically, at test time, they iteratively optimize over the score and scale parameters of Shellhamer et al 2019, where a Gaussian receptive field is used to allow for dynamic scale adaptation of each convolutional layer. They optimize the parameters with respect to an entropy minimization objective. Experiments on PASCAL VOC, reported at multiple scales, demonstrate improvements in IOU over the vanilla architecture. Strengths: -The work was well-motivated -Formulation is pretty elegant and outperforms the baseline Weaknesses: While I liked the somewhat elegant formulation of dynamic test-time scaling proposed by the following work, I don*t think this work introduced many novel results nor insights -Multiscale test-time inference is already standard in state-of-the-art architectures such as DeepLab. Specifically, DeepLab runs inference at multiple scales, then max pools the logit responses across all scales. (see Table 3 of https://arxiv.org/pdf/1511.03339.pdf) -Using entropy as a measure of network uncertainty is a good idea, but also not a novel finding. -COCO and Cityscapes would probably have been better choices for datasets with larger variations in scale Improvements: -Try comparing against the multiscale logit-max-pooling inference procedure as a baseline -- or demonstrate that the proposed technique can further improve upon the results of the logit-pooling technique. -Some details weren*t very well explained. Specifically, what is the task parameter for? ** Post Rebuttal Response I*d like to thank the authors for clarifying some points in their response. Overall, I maintain that I think the optimization-based scale inference solution they present is interesting from an implementation standpoint, but the findings in this work did not yield sufficient new insight for the task. While I agree that their approach is fairly different from common approaches such as discrete image pyramids, a thorough quantitative comparison of these differences would make this work a lot stronger. As such, I maintain my original rating.
assistant
Summary: The following work proposes a test-time optimization over scales to improve semantic segmentation. Specifically, at test time, they iteratively optimize over the score and scale parameters of Shellhamer et al 2019, where a Gaussian receptive field is used to allow for dynamic scale adaptation of each convolutional layer. They optimize the parameters with respect to an entropy minimization objective. Experiments on PASCAL VOC, reported at multiple scales, demonstrate improvements in IOU over the vanilla architecture. Strengths: -The work was well-motivated -Formulation is pretty elegant and outperforms the baseline Weaknesses: While I liked the somewhat elegant formulation of dynamic test-time scaling proposed by the following work, I don*t think this work introduced many novel results nor insights -Multiscale test-time inference is already standard in state-of-the-art architectures such as DeepLab. Specifically, DeepLab runs inference at multiple scales, then max pools the logit responses across all scales. (see Table 3 of https://arxiv.org/pdf/1511.03339.pdf) -Using entropy as a measure of network uncertainty is a good idea, but also not a novel finding. -COCO and Cityscapes would probably have been better choices for datasets with larger variations in scale Improvements: -Try comparing against the multiscale logit-max-pooling inference procedure as a baseline -- or demonstrate that the proposed technique can further improve upon the results of the logit-pooling technique. -Some details weren*t very well explained. Specifically, what is the task parameter for? ** Post Rebuttal Response I*d like to thank the authors for clarifying some points in their response. Overall, I maintain that I think the optimization-based scale inference solution they present is interesting from an implementation standpoint, but the findings in this work did not yield sufficient new insight for the task. While I agree that their approach is fairly different from common approaches such as discrete image pyramids, a thorough quantitative comparison of these differences would make this work a lot stronger. As such, I maintain my original rating.