The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks. However, this summarization tends to lose the intricacies of the per-task curves and it isn't able to reflect if all the tasks are at their validation optimum even if the summarized curve might be. In this work, we explore this loss of information, how it affects the model at testing and how to detect it using interval plots. We propose two techniques as a proof-of-concept of the potential gain in the test performance when per-task validation curves are accounted for. Our experiments on three large datasets show up to a 2.5% increase (averaged over multiple trials) in the test accuracy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum. This potential increase is not a result of any modification to the model but rather at what point of training the weights were selected from. This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics.
The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task.
At different points of training, a model's performance on a task will vary. To make it easier to examine this, they propose displaying an interval plot. The validation accuracy for a single task will be largest at a specific epoch. For each single task, display the epochs where performance is within some threshold of the best validation-set accuracy for that task. Plots on CIFAR-100 demonstrate how some tasks are learned quickly then forgotten, while other tasks are only learned late.
To examine the difference in performance, they compare the single best CIFAR-100 model, to the classifier defined by "for class k, forward input to the model with best validation accuracy for class k". This gives about +2.5% in Top-1 performance, a similar story appears on the other 2 datasets they try (Tiny ImageNet, PadChest). This requires N models, where N is the number of classes. They also examine a clustering approach, where they cluster the N models into K models (using K-means), then forward class k to the best model out of those K models.
Overall, it isn't an especially large contribution but I felt it had some interesting points. Some comments:
* Citation list feels short. The catastrophic forgetting literature seems relevant here. Rich Caruana's multitask learning thesis (Caruana et al 1997) is also relevant.
* It's unclear how the K-means is carried out. I assume the features used for each model checkpoint are tied to its performance on individual task but it's never spell out what distance metric is used in the K-means.
* It feels strange to have no comparisons to ensemble-based baselines. The baseline here would be, for some parameter K, run K independent training runs, take the top average validation accuracy from each run, and average them together, then compare this to the K models found from K-means clustering. For small enough K (like 5 or 10) this seems like a feasible experiment, computation-wise.
* In a similar vein, I wonder how this compares to the Snapshot Ensembles paper, which has a similar guarantee of doing 1 training run and giving M models for an ensemble. Is the gain from ensembling, or from directly examining which models are better at which classes?
* A natural follow-up here is to use model distillation to distill the N best models for each individual class into a single model checkpoint - does this give us a better single model checkpoint?
As-is, I give this paper a weak reject, but would be willing to increase if some of these experiments were tried.
Edit: having read the other reviews and author comments, I still maintain a weak reject rating. I believe this paper needs further work.