ADASAMPLE: ADAPTIVE SAMPLING OF HARD POSITIVES FOR DESCRIPTOR LEARNING
Triplet loss is commonly used in descriptor learning, where the performance heavily relies on mining triplets. Typical solution to that is first picking pairs of intra-class patches (positives) from the dataset to form batches, and then selecting in-batch negatives to construct triplets. For high-informativeness triplet collection, researchers mainly focus on mining hard negatives in the second stage, while they pay relatively less attention to constructing informative batches, i.e., matching pairs are often randomly sampled from the dataset. To address this issue, we propose AdaSample, an adaptive and online batch sampler, in this paper. Specifically, we sample positives based on their informativeness, and formulate our hardness-aware positive mining pipeline within a novel maximum loss minimization training protocol. The efficacy of the proposed method is demonstrated in several standard benchmarks, in which it results in a significant and consistent performance gain on top of the existing strong baselines. The source code and pretrained model will be released upon acceptance.
The paper proposes to make two modifications in hard negative mining procedure for descriptor learning: - instead of selecting hardest samples in a minibatch it proposes to sample proportionally to distance between descriptors - gradient wrt model parameters is weighted inverse proportionally to the distance The authors attempt to analyze the method theoretically and evaluate on two descriptor learning benchmarks, UBC phototour 2011 and HPatches 2017. I propose reject mainly for the following reasons: 1) dataset selection and quality of experimental evaluation 2) the writing (presentation of the results) Regarding (1), my main concern is on statistical significance of presented experiments. The authors conduct each experiment once and do not take into account variance coming from neural net training. For example, in table 2 all numbers seem to be very close so it is not possible to conclude anything. Same is true for other experiments, one way to improve would be to do 10-fold cross validation or at least train 5 network with different random seed and provide mean +- std of the results. Then, I think UBC results are not indicative of performance since the dataset is small and very old at this point, so I disagree with the claim that improvement of ~0.1% is of significant margin. Improvement on HPatches seems to be slightly more significant, but it is not clear wether the authors used comparable networks in terms of number of parameters in the network and descriptor size. Regarding (2), the way the paper is written is very confusing and should be improved. The attempts to have a theoretical derivation of the proposed method are welcome, but should not precede the actual explanation of the method. I suggest to explain clearly the modifications to sampling strategy, and only then relate them to informativeness intuition. Further, the improvement over HardNet is not clear. The two proposed modifications are not tested separately. To have a proper ablation study, the authors could sample negatives proportionally to descriptor distance and remove gradient weighting, and vice versa, to sample hardest-in-batch and only add gradient weighting. I also disagree that hard negative mining is *widely used in object detection and other vision tasks* (quoting the authors). It used to be common in metric learning for face recognition, as the authors cite FaceNet 2015, but has been long abandoned in favor of SphereFace-like approaches. Same is true for object detection, where Fast R-CNN and Mask R-CNN without HNM are widely used.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
ADASAMPLE: ADAPTIVE SAMPLING OF HARD POSITIVES FOR DESCRIPTOR LEARNING
Triplet loss is commonly used in descriptor learning, where the performance heavily relies on mining triplets. Typical solution to that is first picking pairs of intra-class patches (positives) from the dataset to form batches, and then selecting in-batch negatives to construct triplets. For high-informativeness triplet collection, researchers mainly focus on mining hard negatives in the second stage, while they pay relatively less attention to constructing informative batches, i.e., matching pairs are often randomly sampled from the dataset. To address this issue, we propose AdaSample, an adaptive and online batch sampler, in this paper. Specifically, we sample positives based on their informativeness, and formulate our hardness-aware positive mining pipeline within a novel maximum loss minimization training protocol. The efficacy of the proposed method is demonstrated in several standard benchmarks, in which it results in a significant and consistent performance gain on top of the existing strong baselines. The source code and pretrained model will be released upon acceptance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes to make two modifications in hard negative mining procedure for descriptor learning: - instead of selecting hardest samples in a minibatch it proposes to sample proportionally to distance between descriptors - gradient wrt model parameters is weighted inverse proportionally to the distance The authors attempt to analyze the method theoretically and evaluate on two descriptor learning benchmarks, UBC phototour 2011 and HPatches 2017. I propose reject mainly for the following reasons: 1) dataset selection and quality of experimental evaluation 2) the writing (presentation of the results) Regarding (1), my main concern is on statistical significance of presented experiments. The authors conduct each experiment once and do not take into account variance coming from neural net training. For example, in table 2 all numbers seem to be very close so it is not possible to conclude anything. Same is true for other experiments, one way to improve would be to do 10-fold cross validation or at least train 5 network with different random seed and provide mean +- std of the results. Then, I think UBC results are not indicative of performance since the dataset is small and very old at this point, so I disagree with the claim that improvement of ~0.1% is of significant margin. Improvement on HPatches seems to be slightly more significant, but it is not clear wether the authors used comparable networks in terms of number of parameters in the network and descriptor size. Regarding (2), the way the paper is written is very confusing and should be improved. The attempts to have a theoretical derivation of the proposed method are welcome, but should not precede the actual explanation of the method. I suggest to explain clearly the modifications to sampling strategy, and only then relate them to informativeness intuition. Further, the improvement over HardNet is not clear. The two proposed modifications are not tested separately. To have a proper ablation study, the authors could sample negatives proportionally to descriptor distance and remove gradient weighting, and vice versa, to sample hardest-in-batch and only add gradient weighting. I also disagree that hard negative mining is *widely used in object detection and other vision tasks* (quoting the authors). It used to be common in metric learning for face recognition, as the authors cite FaceNet 2015, but has been long abandoned in favor of SphereFace-like approaches. Same is true for object detection, where Fast R-CNN and Mask R-CNN without HNM are widely used.
assistant
The paper proposes to make two modifications in hard negative mining procedure for descriptor learning: - instead of selecting hardest samples in a minibatch it proposes to sample proportionally to distance between descriptors - gradient wrt model parameters is weighted inverse proportionally to the distance The authors attempt to analyze the method theoretically and evaluate on two descriptor learning benchmarks, UBC phototour 2011 and HPatches 2017. I propose reject mainly for the following reasons: 1) dataset selection and quality of experimental evaluation 2) the writing (presentation of the results) Regarding (1), my main concern is on statistical significance of presented experiments. The authors conduct each experiment once and do not take into account variance coming from neural net training. For example, in table 2 all numbers seem to be very close so it is not possible to conclude anything. Same is true for other experiments, one way to improve would be to do 10-fold cross validation or at least train 5 network with different random seed and provide mean +- std of the results. Then, I think UBC results are not indicative of performance since the dataset is small and very old at this point, so I disagree with the claim that improvement of ~0.1% is of significant margin. Improvement on HPatches seems to be slightly more significant, but it is not clear wether the authors used comparable networks in terms of number of parameters in the network and descriptor size. Regarding (2), the way the paper is written is very confusing and should be improved. The attempts to have a theoretical derivation of the proposed method are welcome, but should not precede the actual explanation of the method. I suggest to explain clearly the modifications to sampling strategy, and only then relate them to informativeness intuition. Further, the improvement over HardNet is not clear. The two proposed modifications are not tested separately. To have a proper ablation study, the authors could sample negatives proportionally to descriptor distance and remove gradient weighting, and vice versa, to sample hardest-in-batch and only add gradient weighting. I also disagree that hard negative mining is *widely used in object detection and other vision tasks* (quoting the authors). It used to be common in metric learning for face recognition, as the authors cite FaceNet 2015, but has been long abandoned in favor of SphereFace-like approaches. Same is true for object detection, where Fast R-CNN and Mask R-CNN without HNM are widely used.