Semi-supervised learning (SSL) is a study that efficiently exploits a large amount of unlabeled data to improve performance in conditions of limited labeled data. Most of the conventional SSL methods assume that the classes of unlabeled data are included in the set of classes of labeled data. In addition, these methods do not sort out useless unlabeled samples and use all the unlabeled data for learning, which is not suitable for realistic situations. In this paper, we propose an SSL method called selective self-training (SST), which selectively decides whether to include each unlabeled sample in the training process. It is also designed to be applied to a more real situation where classes of unlabeled data are different from the ones of the labeled data. For the conventional SSL problems which deal with data where both the labeled and unlabeled samples share the same class categories, the proposed method not only performs comparable to other conventional SSL algorithms but also can be combined with other SSL algorithms. While the conventional methods cannot be applied to the new SSL problems where the separated data do not share the classes, our method does not show any performance degradation even if the classes of unlabeled data are different from those of the labeled data.
This paper describes the method for performing self-training where the unlabeled datapoints are iteratively added to the training set only if their predictions by the classifier are confident enough. The contributions of this paper are to add datapoints based on the prediction of the confidence level by a separate selection network and a number of heuristics applied for better selection. On the experimental side, the contribution is to test the scenario where datapoints from irrelevant classes are included in the unlabeled dataset.
The paper is written in a way that makes following it a bit difficult, for example, the experimental setups. Also, the writing can be improved by making the writing more concise and formal (examples of informal: "spoil the network", "model is spoiled", "problem of increased classes", "many recent researches have been conducted", "lots of things to consider for training", "supervised learning was trained" etc.). The contributions of the method could also be underlined more clearly in the abstract and introduction. The description of consistency regularisation methods in section 2.2 is not very clear and I would like to get better understanding of temporal ensembling and SNTG methods here as they play an important role in the experiments. 
The idea of selective sampling for self-training is promising and the investigated questions are interesting. As far as I understand, the main contribution of this paper is the use of separate "selection network" to estimate the confidence of predictions by "classification network". However, as the "selection network" uses exactly the same input as "classification network", it is hard to imagine how it can learn additional information. For example, imagine the case of binary classification. If the selection network predicts 0 in come cases, it can be used to improve the result of "classification network" by flipping the corresponding label. How can you interpret such a thought experiment? One could understand the use of "selection network" as a way to automatically select a threshold of what to consider confident, however, in this case, the prediction of "selection network" should be thresholded at 0.5 (correct prediction or not), but the experiments use complex thresholds. Could you elaborate more on why the selection network is needed? How would it compare to a simple strategy of only including the datapoints whose top-1 prediction of "classification network" is greater than some threshold? Finally, could you show a plot of top-1 prediction of "classification network" vs score of "selection network" and elaborate on that?
Then, in sections 3.2 and 3.3 the authors introduce a few additional tricks for self-training: exclude datapoints whose predictions are changing and balance the classes. Intuitively, these criteria are well motivated, but unfortunately, the combination of all the intuitions (including "selection network" with threshold) is not very principled. Ablation study shows that the use of the "selection network" strategy does not improve the results without these heuristics. It would be interesting to see how these heuristics would do without "selection network", for example, either by doing simple self-training with thresholding on the score of the classifier or by applying only these heuristics in combination with TempEns+SNTG. In the current form of evaluation, it is hard to say if there is any benefit of using the "selection network" that is the main novelty of the paper.
It is very valuable that the experimental results include many recently proposed methods. Besides, the settings are described in details that could help for the reproducibility of the results. However, I have a few concerns about the results. First of all, the proposed SST algorithm alone only performs better than baselines in 1 case, equal to them in 1 case and worse in 1 (table 3). Besides, as the base classifier is different for various baselines, it is hard to compare the methods. Then, the important hyperparameter of the method---threshold---seems to be hard to select (both in sections 4.1 and 4.2). How did you chose the current values? How sensitive is it? Why various datasets need different settings? How the threshold value can be set in practice? Another important parameters is the number of iterations of the algorithm. How was it chosen? Concerning the experiments of section 4.2, how would the baseline methods of section 4.1 do in this case? Why did you select to study animal vs non-animals sets of classes? What would happen if you use random class splits or split animal classes (like in a more realistic scenario)? 
To conclude, while I find the studied problem quite interesting and intuitions behind the method very reasonable, the current methodology is not very principled and the experiment evaluation did not convince me that such an elaborate strategy is needed.
Some questions and comments:
- The setting of including unrelated classes in the unlabeled data resembles transfer learning setting. Could you explain why the ideas from transfer learning are not applicable in your case?
- In the training procedure of "selection network" of Sections 3.1, do you use the same datapoints to train a "classification network" and "selection network"? If it is the case, how do you insure that the "classification network" does not learn to fit the data perfectly and thus all labels s_i are 1?
- In the last sentences of the first paragraph on p.2 you make a contrast between using softmax and sigmoid functions, however, normally the difference between them is their use in binary or multiclass classification. Is there anything special that you want to show in you case?
- What do you mean in section 3.3 by "if one class dominates the dataset, the model tends to overfit"?
- I think parameters of training the networks from the beginning of section 4 could be moved to the supplementary materials.
- Figure 3: wouldn't the plot of accuracy vs amount of data be more suitable here?
- Synthetic experiments of supplementary materials: the gains of the methods seem to be small. What are the numerical results? What would happen if you allow to select starting point at random (a more realistic case)?
- Can you explain the sentence "To prevent data being added suddenly, no data was added until 5 iterations"?
- How was it possible to improve the performance in experiment of section 4.2 with 100% of irrelevant classes?