We present SOSELETO (SOurce SELEction for Target Optimization), a new method for exploiting a source dataset to solve a classification problem on a target dataset. SOSELETO is based on the following simple intuition: some source examples are more informative than others for the target problem. To capture this intuition, source samples are each given weights; these weights are solved for jointly with the source and target classification problems via a bilevel optimization scheme. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting. SOSELETO may be applied to both classic transfer learning, as well as the problem of training on datasets with noisy labels; we show state of the art results on both of these problems.
PROS:
* This is an interesting approach of assigning contribution weights to each source sample.
* Could be very helpful for tasks where we have a noisy and a (small) clean dataset.
* The method seems to be performing well for the tasks chosen, especially for the CIFAR experiments.
* Simple idea and relatively easy to implement
CONS:
* Clarity could be improved, especially in the experimental section
* The motivation for the SVHN 0-4 to MNIST 5-9 is not clear. It would make more sense to me to transfer between SVHN 0-5 to SVHN 5-9, or from the entire SVHN to the entire MNIST, but this particular transfer seems somewhat irrelevant to the claims. The two domains are particularly dissimilar and trying to select "good" SVHN samples according to 20 or 25 MNIST samples seems somewhat ill-posed. It is also particularly surprising to me that 25 MNIST samples were enough to train a LeNet to the point of 84% accuracy on the entire MNIST test set. (I'm referring to the target-only line) Is that really the case, or was a larger training set used for that particular line?
* There is a claim that "SOSELETO has superior performance to all of the techniqiues which do not use unlabelled data", however I'm not sure whether these techniques were used as prescribed and if the comparison was fair. For example, I believe domain adaptation techniques like DANN, largely assume a common label space between the domains.
* Comparison with previous re-weighting techniques would have been very informative.
QUALITY:
* The quality of the writing was overall high, with a few exceptions, including the related work and the experimental section.
* In related work, the "bilevel optimization" section could be a bit more descriptive, maybe some of the explanationgiven in Sect. 3 could be moved here?
* The experiments were convincing, with the exception of the SVHN to MNIST section.
CLARITY:
* I believe a better synthetic experiment could be chosen to highlight the approach: how about a truly noisy dataset that is not as separable as the "noisy" dataset in Figure 1? Maybe you could have the same noisy dataset but with a small portion of random points having the wrong label. For the same experiment, it should be clearly stated that your task is binary classification and what was the classifier used.
* For the CIFAR experiments, it is very good that it performs well, but it'd be informative to see if SOSELETO can perform even better with 10K samples.
* It wasn't clear to me whether the a-values of only one batch (32 samples?) at a time were affected. If so, how does this scale to really large datasets like, say, Imagenet?
* In the CIFAR experiments, it is mentioned that a target batch-size is chosen to be larger to enable more a-values to be affected. This seems like a typo, but it was confusing. (I assume that the source batch-size is chosen to be larger)
* Figure 2 could use a better caption and a legend. It would also be an easier figure to parse if the x-axis was reversed (eg. if the x-axis was the fraction of data used)
* It was not clear to me what "true transfer learning" means as opposed to domain adaptation.
ORIGINALITY:
* It seems that this idea has been explored before, however I'm not personally familiar with that work. I would have definitely liked to see comparisons with it though.
SIGNIFICANCE:
* This is a simple idea that seems to work well. As I wrote above, it would be great to know how it compares to other re-weighting techniques.