extfieldsofstudy:
Medicine
Computer Science

text:
Taming Overconfident Prediction on Unlabeled Data from Hindsight

Minimizing prediction uncertainty on unlabeled data is a key factor to achieve good performance in semi-supervised learning (SSL). The prediction uncertainty is typically expressed as the \emph{entropy} computed by the transformed probabilities in output space. Most existing works distill low-entropy prediction by either accepting the determining class (with the largest probability) as the true label or suppressing subtle predictions (with the smaller probabilities). Unarguably, these distillation strategies are usually heuristic and less informative for model training. From this discernment, this paper proposes a dual mechanism, named ADaptive Sharpening (\ADS), which first applies a soft-threshold to adaptively mask out determinate and negligible predictions, and then seamlessly sharpens the informed predictions, distilling certain predictions with the informed ones only. More importantly, we theoretically analyze the traits of \ADS by comparing with various distillation strategies. Numerous experiments verify that \ADS significantly improves the state-of-the-art SSL methods by making it a plug-in. Our proposed \ADS forges a cornerstone for future distillation-based SSL research.

I. INTRODUCTION
L EARNING with partially labeled data implicitly requires the model to exploit the missing labels on its own. This form of learning paradigm, known as semi-supervised learning [1] (SSL), is of practical significance as labeling costs are now passed on to the subsequent algorithm design.
The success of modern SSL algorithms is essentially attributed to two indispensable components designed for unlabeled data. (1) Consistency regularization. It assumes that each unlabeled sample u should have the consistent prediction with its transformed counterpart T (u). This self-supervised constraint on unlabeled data extends the conventional similarity regularization over neighboring samples [2]- [4], providing an favourable optimization direction for classification, which thus is greatly boosted by recent data augmentation techniques [5], [6]. (2) Prediction distillation. The predictions on unlabeled data are often of high uncertainty due to the absence of supervised information. Prediction distillation tries to enhance model prediction based on current output, e.g., following the principle of Minimum Entropy [7]. As a consequence, the decision boundary is encouraged to deviate from the region where the ambiguous predictions exist. Many SSL models [7]- [10] in literature have benefited from this component. The  Generally, given a labeled set L and an unlabeled set U, a SSL model parameterized by θ typically minimizes the following objective where J S denotes a supervised loss on labeled set L, J C and J D denote the consistency loss and distillation loss on unlabelled data U, respectively. α and β are non-negative weight factors for balancing three loss terms. Despite the great boost of consistency regularization to SSL, extra expertise is usually required to customize the effective data augmentation w.r.t. different data types [11]. Conversely, prediction distillation manages to learn from the model output, serving as a more general and practical tool in the machine learning community. Besides the aforementioned Minimum Entropy (ME), many other distillation strategies and their variants have been developed lately, such as Sharpening (SH) [12]- [14], Pseudo-Labeling (PL) [15]- [17], Negative Sampling (NS) [18]. From the optimization perspective, although many of them progressively encourage partial unlabeled data to reach the low-entropy state, existing strategies unavoidably introduce the incorrect distillations because neural networks are argued not well calibrated [19]. The corresponding predictions are dubbed as overconfident predictions 1 in this paper. That means for unlabeled predictions, not all distillations are beneficial. Many recent studies can be interpreted as workarounds to this problem. By discriminating the importance of unlabeled data, [20] learns the sample-wise weights for unlabeled data through additive labelled validation set [20]. A more direct thought is leveraging the self-paced technique which selects the promising unlabeled samples [21] to do optimization in each iteration. Lately, [22] proposes to reduce the bias of PL by approximating the calibration criterion from the overall uncertainty. There methods practically require a lot of extra efforts, albeit some improvement are made. Specifically, the labelled validation set needed by [20] is hard to get in real world. Self-paced strategy [21] may slow down the convergence rate, whose reported performance cannot rival the stateof-the-arts yet in their paper. The approximated calibration executed in [22] is implemented by Monte Carlo dropout [23] which demands multiple times inferences more than standard SSL training. Therefore, instead of following this branch of works, our research scope restricts to remedying overconfident predictions via tailor-designing more intelligent prediction distillation strategy which is expected to be less computational and more extendable.
Looking back on the execution of a SSL model which is equipped with a distillation component, we realize the overconfident prediction problem could be remedied if we stop greedily encouraging predictions to be one-hot and use the informative ones instead. With this hindsight, we present ADaptive Sharpening (ADS), a novel and simple prediction distillation method. By replacing the activation function with sparsemax transformation, ADS performs adaptive selection on the logits layer, which prepares for the consequent loss computation. As a result, ADS concentrates on a set of promising classes automatically selected for each unlabeled sample and avoids the unnecessary back-propagation for non-target class as well. Additionally, ADS constructs a mild target label distribution by exponentially raising the sparse probabilities, leading to an informative optimization for minimizing prediction uncertainty. We borrow the name of Sharpening used in [13], [24] to our method, but ADS is essentially distinct from SH according to the comparison of Table 1. We theoretically analyze the working mechanism of ADS and demonstrate its superiority by numerous experiments. The major contributions of this paper are summarized as: 1) We propose a new distillation strategy, named ADaptive Sharpening (ADS), to distill low-entropy predictions for unlabeled data. The proposed ADS is simply used a plugin for SSL algorithms. 2) We analyze the connection and difference between ADS and other distillation strategies through extensive comparisons. In particular, ADS adaptively masks out the overconfident and negligible predictions and promotes the informed predictions only which is more instructive. 3) We verify the superiority of our ADS by combining it with advanced SSL models, including Virtual Adversarial Training (VAT) [9], MixMatch [13], and FixMatch [16]. The experimental results on different benchmarks demonstrate ADS achieves significant improvements over existing technologies. This paper is organized as follows. In Section II, we present a series of recent SSL works that are closely related to our study. Section III revisits various distillation strategies which are analyzed from entropic view. The proposed ADS and its corresponding SSL training model is introduced in Section IV, followed by the theoretical justification for its properties in Section V. Experiments in Section VI demonstrated the efficacy of ADS. Section VII concludes the whole work and the last is the appendix.

II. RELATED WORK
As there is a vast literature in SSL, we focus on the most related works which are expressed from the following two perspectives.
Consistency Regularization The concept of consistency can date back to the graph-based SSL work [2] where similar training pairs are expected to have same label assignment. Recent SSL research has extended consistency to a more general case where natural unlabeled data and their transformed counterparts should correspond to the same output. This is closely related to the thought of self-supervised learning and many SSL studies [12], [14], [25], [26] are furthering this direction. The adopted transformations play an important role therein, and they may include multiple passes of a single input with adversarial perturbations [27], randomized data augmentations [5], [6], etc. Beyond the above consistency, using the prediction of weak augmentation to induce the prediction of corresponding strong augmentation [16] lately shows superior performance. Summarily, consistency regularization is helpful because it facilitates the model to be insensitive to the possible disturbances, which is exactly desired for classification tasks.
Prediction Distillation Existing research approaching the certain predictions over unlabelled data typically adopts a post-processing distillation [28]. The pioneer strategy, such as ME [7] and its variant [25], plainly punishes the high entropy predictions, enabling the model to have a decision boundary that traverses low-density regions. Thus, SSL models equipped with ME will exactly have one-hot prediction for all unlabelled data in theory. Differently, SH [24] constructs a mild target to align with in each step, showing a good performance in MixMatch [13]. PL [15] is well-known and widely applied in recent research [16], [17], [21]. Opposite to above approaches, NS [18] lately resorts to minimizing the predictions of the classes the sample probably does not belong to. It is viewed as a safer option since searching for the negative classes based on the model output introduces less bias than directly picking up the true class. Notation All the vectors are written in boldface. For a vector p, the k-th entry is denoted as p i , and p (k) is the k-th sorted coordinate of p in an descending order. The classification network with the model parameters of θ is denoted by f (; θ), whose output logits vector is denoted by z.

III. PREDICTION DISTILLATION STRATEGY REVISIT
In this section, we first present the formulation of existing distillation strategies and then analyze how they work from entropic view.

A. Formulation
For a logits vector z, softmax function produces strictly positive predictions by componentwise computing For the better statement, we further denote p = softmax(z). Now we formulate every distillation strategy as follows. ME The loss for minimum entropy of an unlabelled sample is written as Notice that ME is originally defined on unlabelled data, and it applies to augmented unlabelled data [9] as well. SH Sharpening function [13], [29] constructs the target label by adjusting the temperature of the current categorical distribution p,  where λ (λ > 0) is the temperature that controls how sharp the output distribution looks like. Then the corresponding distillation loss is formulated as PL Naive Pseudo-Labeling [15] could be described as a hard version of sharpening, and it picks the class which has the maximum predicted probability to be the target label, In practice, a stricter condition applies [16] that requires the maximum probability to exceed a predefined threshold τ PL . This means that PL does sample selection in each round optimization. For each selected sample, the distillation loss is written in a similar manner with SH, NS Picking negative classes by a threshold τ NS and minimizing their probabilities comes to the distillation loss of where I(·) is the indicator function. This loss term is proved to better approximate the true likelihood of unlabeled data in [18] and we interpret it as one of the distillation baselines.

B. Entropic View of Distillation
We observe that existing distillation strategies are in pursuit of low-entropy predictions via post-processing the model predictions in different ways. According to their formulations, we summarize them into the left panel of Figure 1 where they are compared in terms of candidate classes selection, the sparsity of activation, and the way of uncertainty minimization. Before diving into more details, we need informally introduce the following definitions.
The labeled data can be viewed as anchor points which initialize the model and induce the unlabeled prediction. Given 0 < θ1 0.5 θ2 < 1, we adhere the philosophy of this thought and partition the intact probability space into four intervals: The above definitions allow us to roughly categorize each prediction as one of the four types. Thus, we can see that ME and SH consider all classes as the candidates and aggressively minimize uncertainties over all categories except ambiguous ones which provide no useful information for distillation. By simply setting θ 1 = τ NS and θ 2 = τ PL , we conclude that PL and NS resort to heuristic thresholds to ensure that distillation acts on relatively certain (determinate or negligible) predictions only. As a result, ME, SH, and PL suffer the problem of overconfident predictions [17], [20] as they are plainly enhancing the determinate predictions (type I in Figure 1). Notice that NS tries to penalize the negligible predictions, which seems to be safer. However, we demonstrate that it factually has the similar effect with PL for binary class classification as further discussed in Section V-A.
For the sake of better understanding to the different distillation strategies, we display the four types of prediction in binary-class classification case on the right panel of Figure 1. As we have interpreted all distillation strategies as uncertainty minimization, we indeed decompose each distillation strategy in this probability space. It is observed ambiguous prediction is special and it only takes a point. θ 1 and θ 2 does not coexist in existing methods, and they are not always empirically satisfying θ 1 +θ 2 = 1. In this work, we aim to leverage the informed predictions only, which remedies the overconfident predictions by doing nothing to the extremely certain predictions (Extra benefits can be referred in Section V-B). Particularly, as in different training stages and for different unlabeled samples, the informed predictions should be relatively different and dynamic. Thus, the informed predictions are not quite aligned with the ones defined by hard thresholds used in PL and NS.

IV. ADS BASED SSL FRAMEWORK
According to the understanding of prediction distillation indicated by the left panel of Figure 1, we propose ADaptive Sharpening (ADS), a dual distillation mechanism which only sharpens the informed predictions so as to avert the overconfident issues. An architecture of ADS from logits to losses is showcased as Figure 2. We also analyze how do other loss terms benefit from ADS, which completes the presentation of concrete ADS based SSL framework. A. ADaptive Sharpening (ADS) 1) Adaptive Selection via Sparsemax: Properly selecting partial classes to do distillation assists the classification model to concentrate on the most confusing classes for each unlabeled sample [18]. Existing prediction distillation strategies, such as PL and NS, are plainly overlaid on the probabilistic distribution produced by softmax function, formulating a hard threshold-based post-processing approach [28]. In spite of their popularity in recent research [16], [18], the major drawback of them is the adoption of a predefined threshold, which is not robust enough to account for individual variability and model variability at different training stages, dwindling its effectiveness in practical implementations.
Therefore, we aim to design a flexible sampler, which can automatically determine a partial set of promising classes as candidates for true class, being adaptive to each unlabelled sample at different training stages. To this end, we study this problem from neural networks' output, i.e., logits, and apply sparsemax [30] to produce a sparse probability distribution directly, instead of applying a truncation over probability values afterward. For a neural networks' output logits vector z, sparsemax is calculated by where [u] + := max(0, u). τ (z) serves as a soft threshold, which is briefly expressed as follows. Let . Basically, given a logits vector z, sparsemax finds a threshold τ (z) and sets probabilities below τ (z) to zeros while satisfying j [z j − τ (z)] + = 1. k(z) indicates the support of sparsemax(z) which makes τ (z) fall in the "maximum gap" of the sorted logit values. We are in favor of this valuable trait in SSL as sparsemax adaptively rules out some unrelated classes, which is viewed as more flexible and less biased compared with those using a hard threshold.
Selecting proper candidates motivates the employment of sparsemax. This process also implies the informed predictions desired by our distillation component. We focus on the pipeline of our method here and leave the analyses how ADS fulfill our hindsight in Section V-A.
2) Sharpening on Sparse Probabilities: Prediction distillation acts as a self-training process. To iteratively refine the classification model, we define the distillation loss of ADS as a distance, e.g., Kullback-Leibler (KL) divergence, between the raw prediction p and the auxiliary target label distribution q: where p i = sparsemax i (f (u; θ)). The choice of target label distribution q is crucial for ADS's performance. Particularly, we would like our target label distribution q to have the following properties: (1) strengthen confident predictions, (2) exclude the contributions of negative class predictions. According to the comparison in Figure 1, Sharpening is a feasible option to satisfy the above requirements. Having a sparse prediction p in hand, to construct the according target label distribution q, we first exertp = sg(p), where sg stands for the stop-gradient operator that is defined as identity during forward computation and has zero partial derivatives. After that, similar to Eq. (4), we calculate every coordinate q i of q as a normalized probability raised fromp i with a power of r: Obviously, ifp i = 0, we have q i = 0, and ifp = 1 K 1, we have q =p. Otherwise, as r → ∞, the categorical distribution q will approach a delta distribution. We emphasize that although ADS borrows the idea of SH for constructing the target label distribution, they have the different functionalities during distillation. Particularly, we will show ADS does not aggressively distill relatively certain predictions in Section V-A, which mitigates the overconfident risk. Figure 2 shows the process of distillation design starting from derived logits. Note that although Eq. (10) is defined on unlabeled data, it could be extended to augmented unlabeled instances in some methods. Accordingly, the augmented data will pass the sparsemax once they are expected to participate the distillation loss.

B. In Conjunction with Other Loss
As supervised loss and consistency regularization are both computed over predictions, unlike other distillation strategies ADS indeed has an direct impact on these two losses as well.
In terms of consistency regularization, it restricts each unlabeled sample to reach a consistent prediction with its transformed counterpart. For ADS, consistency loss J C could be generally formulated as where z and z are logits vectors of the inspected unlabeled sample and its counterpart, and Dist(·) could be any proper distance function. Suppose sparsemax i (z) = sparsemax i (z ) = 0 holds for the i-th class. In this case, class i will not contribute to the consistency loss. With more classes like this, the consistency loss of ADS will focus on a few of confusing classes only. From Eq. (2), the condition of equality of softmax i (z) = softmax i (z ) is much stricter. That means softmax-based consistency will be distracted by the unrelated classes during model training. Taking image classification for an example, an unlabeled image is categorized to leopard only if all its transformed counterparts reach a consensus on leopard. In practice, the improvement of consistency regularization essentially lies in its constantly confirmed decision among the confusing classes. Intuitively, an augmented leopard image might be somehow close to cat while it should less similar to dog. Thus, putting more attentions on confusing classes are expected to achieve better performance which is also referred in previous research [23]. Particularly, we also allow the networks' output of labeled data to pass the sparsemax. However, if the model assigns zero probability to gold label, the entire training sample would be ruled out [30]. Although there exists some possible workaround like adding a small constant to the probabilities and then do re-normalization, we instead use the following supervised loss as an alternative of cross entropy proposed by the recent work [31] where y is the groundtruth encoded by one-hot format. Notice that the 2 -norm based loss trickily averts the aforementioned optimization dilemma, and the derived gradient w.r.t. z has the closed form, making a friendly backward propagation.

V. THEORETICAL ANALYSES
In this section, we theoretically justify the principle of ADS, and then provide the evidence of its superiority from the view of transformation function.

A. Promoting Informed Predictions
We highlight that ADS does distillation via promoting the informed predictions only, showing an efficient distillation mechanism compared with existing strategies.
Regarding a two-class case, we denote the neural networks' output as z = (u, 0). In terms of same z, suppose softmax 1 (z) = s and sparsemax 1 (z) = s . In particular, we have s = s if u = 0.5. Since existing distillation strategies work over softmax output, we consider s as a function of s and rewrite the distillation loss J D (refer to the detailed derivation in Appendix A), making a clear and direct comparison for all methods in the same probability space. Figure 3 plots the target probability, prediction distillation loss, and according gradient versus the first dimensional probability s in terms of each distillation strategy, respectively. We remind readers to read this figure from the shape and trend of every curve other than the scale as they can be equipped with different weights during formulations. Our main conclusions are as follows: • ADS is the unique strategy focusing on informed predictions only. According to Figure 3(b), we observe that the proposed ADS has zero penalty when s is negligible or determinate. Notably, ADS is viewed as a corrective SH by masking out the determinate predictions to avoid introducing overconfident risk. • PL and NS are consistent strategies for binary classification case. PL and NS have the complementary philosophy from Figure 3(a). Given proper thresholds, they are shown to have the same form of loss and gradient according to Figure 3(b) and (c). • ME has the unbounded gradients. ME takes the highest loss value but produces zero gradient on ambiguous predictions according to Figure 3(b) and (c). Meanwhile, when s is relatively small or large, the gradient of ME is large, exposed to the risk of unstable optimization.
From above analyses, ADS will outperform other distillation strategies if the assumption holds that leveraging the informed predictions only are sufficient for distillation. From Figure 3 we realize the terminology "informed" is factually specified by a corresponding threshold in the probability space of softmax output. Now we are exploring a more general case where data could be categorized to one of the multiple classes.
Theorem 1. For an unlabeled sample u whose softmax output is p ∈ R K (K > 2), the distillation loss J D ≡ 0 for ADS in Eq. (10) holds if p (1) > ep (2) , where p (1) , p (2) are the first two largest coordinate of p and e is Euler number.
From Theorem 1 we can see that the "informed" predictions spoken of in the output space of softmax refer to the categorical probability p which satisfies p (1) < ep (2) and p = 1 K 1. That means unlike existing distillation strategies, ADS does not encourage the relatively certain predictions to further become extremely certain since the distillation loss J D ≡ 0 holds as long as p (1) ≥ ep (2) , and thus the issue of overconfident predictions are mitigated. The following corollary quantify the soft threshold used by ADS. ρ+e ρ (K−ρ) , e ρ ρ+e ρ ] in the corresponding softmax output space, respectively, where e is Euler number and ρ is the population of non-zero predictions.
We leave the proof of this corollary to Appendix B. Notably, we clarify that the determinate predictions are spoken of in the context of multi-class classification where only a single class is the groundtruth. In other words, a prediction is said determinate in terms of its potential to be the real label. In addition, the negligible predictions are handled in a similar manner, but its population for a single unlabelled sample is at least 1. In particular, for ADS the negligible predictions do not coexist with the determinate prediction.

B. Sparsemax Facilitating Entropy Minimization
Entropy minimization is a golden principle that has been demonstrated effective in existing SSL research [7], [9], [10]. In this section, we point out this principle is fundamentally related to the probability transformation function used in neural networks, e.g., softmax activation. It typically involves projecting a logits vector z on the probability simplex with an optimized problem of p = arg min where ∆ K−1 is the probability simplex with freedom of K −1, and −H(p) is a convex function, serving as a regularizer.
When H(p) is implmented using Shannon entropy, i.e., H(p) = − k p i log p i , the closed-form solution of Eq. (17) is the softmax transformation, a.k.a. maximum entropy transformation. When H(p) is replaced with Gini entropy, namely H(p) = 1 2 k p i (1−p i ), the solution of Eq. (17) is equivalent to the sparsemax transformation, i.e., Eq (9).
Back to the minimum entropy principle, existing distillation methods built on softmax can be viewed as minimizing the prediction entropy in a post-processing manner [28], which is in conflict with the function of the regularizer H(p) in Eq. (17). As Shannon entropy is a stronger penalty than Gini entropy, softmax would intensify contradictions compared with sparsemax [31]. Therefore, we argue that our ADS facilitates entropy minimization since it reaches a better trade off between probability transformation Eq. (17) and predication distillation J D .

A. Experimental Setup
We conduct empirical comparisons following the experimental setting of [18] except FixMatch for which we adopt the codebase from [16]. For fair comparison, the default network architecture for all datases except MNIST is Wide ResNet-28-2 [32] with 1.5M parameters. Regarding MNIST, we employ a 7-layer convolutional neural network. The batch size for unlabeled data is 64. In terms of the labeled data, the batch size is set as the number of labeled data if it is smaller than 64, and set as 64 otherwise. We do not exhaustively adjust the network parameters for different benchmarks, such as the number of scales and filters, in order to best reproduce the results. The mean and variance of five independent running under different random seeds are reported for fair comparison. Throughout all the experiments involving ADS, we use r = 2 as a default setting as we observe this value achives stable performance for different datasets.
In particular, we use "X+Y" to dub a method by adding a distillation strategy Y to a SSL algorithm X, and use "X-Y" to dub a SSL algorithm X whose distillation component is replaced by Y.

B. Study on VAT
Virtual Adversarial Training [27] (VAT) serves as a powerful SSL algorithm without requiring manual data augmentation [10]. It contains a consistency loss which has a similar form with Eq. (12), but optimizes the adversarial perturbation r adv = g ||g|| , where is a predefined perturbation scale and g is approximated by the gradient on a randomly sampled unit vector r, that is VAT is suggested to be improved if it is added to a minimum entropy according to [9]. Based on this fact, we implement various distillation strategies on VAT for a direct and fair comparison. To this end, we construct each baseline by replacing the distillation loss J D of Eq. (1) with differential distillation strategies. Experiments are conducted on MNIST and CIFAR-10 with 20 and 4,000 labeled training examples, respectively. Table I presents the test error of various distillation strategies based on VAT. It is observed that each distillation loss assists semi-supervised learning and obtains the better performance. More importantly, our ADS achieves the lowest test error compared with both vanilla VAT and other variants. In particular, the improvement to the second-best on two datasets is around 4.7% and 1.5%, respectively.
To have a close look at the distillation component for MNIST dataset on which a better improvement has been achieved. For each method we visualize the distillation loss J D and average dominant probability. Average dominant probability is calculated over all unlabeled training samples (1) which reflects the certainty of unlabeled predictions. We record two measures at each epoch and results are shown as Figure 4.
In terms of our ADS, the distillation loss decreases stably with the increase of average dominant probability. The curve of p (1) reaches a cusp and levels off, with more modest increases afterp (1) = 0.91. For the other four strategies, the distillation loss decreases dramatically, and the average dominant probability surges within a small number of epochs. Clearly, their values of p (1) locate at the cusp, which are all much larger than that of ADS 's. That is to say, they aggressively optimize the learning model to an extremely certain state but inevitably introduce overconfident distillations. Instead, our ADS gradually optimizes the model by using the informed predictions, averting the overconfident risks naturally. Let U + and U − denote the correctly and incorrectly classified unlabeled training examples, and we can rewrite average dominant probability asp (1) = 1 For the correct ones, they are expected to be more certain, but are not supposed to be extremely certain in practice. An evidence for this claim is that in supervised task, penalizing the low-entropy prediction has been demonstrated to be beneficial to model generalization. One of the simplest trick is label smoothing [33]. For the incorrect ones, their dominant prediction should be small as they provide the wrong optimization direction. To sum up, the relative smallerp (1) than competitors suggests better performance by considering two aspects. This is another hindsight by reviewing existing methods.

C. Improvement on Advanced SSL Algorithms
To evaluate the efficacy of our ADS on the SSL algorithm which benefits from data augmentations, we plug ADS in two state-of-the-art models MixMatch [13] and FixMatch [16]. Since vanilla MixMactch has already incorporated SH while vanilla FixMatch has adopted PL as their distillation methods separately, we do not conduct the corresponding ablation study. Notably, ME does not construct an explicit target label, and thus cannot be applied to data augmentations based SSL algorithms, which requires distilling explicit labels for augmented unlabeled data. Following [18], we evaluate the remaining baselines on four benchmarks CIFAR-10, CIFAR-100, SVHN, and STL-10 and present the experimental results in Table II. Table II shows that ADS based methods achieve significant improvement over vanilla Mixmatch, Fixmatch, and their variants, demonstrating the superiority of ADS over other distillation strategies, i.e., SH, PL, NS. In addition, we observe that MixMatch+NS also achieves some improvement over vanilla Mixmatch. Interestingly, this method can be viewed as a simple compensation of overusing the extremely certain predictions (See the distillation loss of SH and NS in Figure 3(b)). From this aspect, ADS alone is functionally similar to this combination and MixMatch-ADS is shown to outperform the two baselines. FixMatch-ADS most time improves FixMatch except for CIFAR-10 with 250 labels. This failure case may result from the occasion that PL can deny the strong augmentations that are out of data distribution once its predictions on weak augmentation are not very satisfactory. In a ward, all these observations are consistent with our claim that promoting informed predictions are beneficial for SSL model training to distill confident and reliable output.

D. Safety with Different Backbone Structures
Intuitively, it should be safe to do candidates selection when every class is initialized to be evenly activated, while a concern arises if the neural network is biasedly initialized to prefer some specific classes which fail to cover the true label. To address this concern, we apply two different neural network architectures on CIFAR-10 with 250 training labels as an example. Apart from the default ResNet which is initialized with the normal distribution, CNN13 [34] is employed which is typically initialized with the uniform distribution. We record the average activationsm and Top-m accuracy over all unlabeled training data in Figure 5, where example-wise m is the number of non-zeros entries. Note that "Top-m accuray" denotes the success rate of partial activations including the true label, and it degenerates to the standard accuracy if m = 1. Figure 5(a) shows that CNN13 activates all the classes at the beginning (m = 10) and then rules out unrelated classes till convergence. Contrastingly, the number of activations for ResNet surges from the one-hot state (m = 1) in the first few of epochs, then drops to a low value, gradually reaches to a steady value afterwards. Figure 5 (b) shows that ResNet achieves better performance than CNN13 in terms of both two measurements although it starts with a biased initialization ( Figure 5 (a)), experimentally demonstrating that initialization is not an issue for candidates selection of ADS. In addition, since standard accuracy is upper bounded by Top-m accuracy as we expect, there exists a considerable gap of around 5% for ResNet and 10% for CNN13. This observation suggests that the selected candidates of ADS are often meaningful.
Notably, neither ResNet or CNN13 converges to one-hot format in the above experiment; the converged average activationsm of them are around 3 and 2, respectively. A possible reason for this phenomenon is that the distillation loss of MixMatch is explicitly defined on the augmented unlabeled data despite their qualities instead of the natural ones, and our model rejects to admit any class as the true label for during distillation. A further question is raised that if they are equivalent to the counterparts in which only Top-3 and Top-2 predictions are picked (with the remaining truncated) and distilled during the entire training stage. This thought is also known as set-valued classification [35], and we compare them as a new group of experiment. Figure 5(c) presents the accuracy comparison between ADS and fixed Top-m selection. We can see both two backbones have the better accuracy when ADS is used, where their activations are dynamically changed indicated by Figure 5(a). In addition, it is also observed that Top-m selection suffers the intense fluctuation on accuracy, which is rooted in the unstable optimization of the plain truncation. Particularly, the accuracy drop for CNN13 is severer than ResNet, which suggests another benefit of residual structure.

E. Observation of Prediction Histograms
We further verify the mechanism of ADS by recording the prediction frequency of unlabeled training data on MNIST and CIFAR-10 dataset. To this end, we evenly partition the prediction space into 10 intervals and let each of them serve as a bin to count the predictions whose values fall into it. Each bin is a half-closed interval except the last one. For example, the first bin is defined as [0, 0.1) and the last is [09,1]. The experiments are executed following the settings of Section VI-C where VAT serves as the base model. Figure 6 presents the numerical distribution of prediction values over a series of epochs, where the maximum epoch is 100 because they are found not clearly altered after 100 epochs optimization. For MNIST, the prediction values are around 0.1 for every sample at the beginning. Then they gradually shift to other bins with the process of optimization. Eventually, almost all the predictions fall into the first and last bin, which means the predictions are distilled to be sufficiently certain. Regarding CIFAR-10, the results are a bit different. The initialization shows the model has some preference as mentioned in Section VI-D. In addition, the predictions grouped into the last bin are relatively fewer than MNIST. This is because the discriminative information for CIFAR-10 images is harder to capture via unlabeled data. In other words, given a semi-supervised model and training data, ADS tries to utilize the informed predictions and leaves the unreliable samples underfitted.

F. Ablation Study
Our ADS consistents of two main components: sparsemax for candidates selection, and Sharpening for enhancing the sparse activations. In this section, we study their independent impact using CIFAR-10 with all but 4,000 labeled data. Particularly, we replace ADS with different ablations, and combine each of them with VAT and FixMatch, respectively. Table III shows that: (1) ADS with default parameter r = 2 achieves the best performance, without extra efforts for fine-tuning regarding different datasets. (2) sparsemax solely imporves the basic SSL algorithms because it constrains the consistency loss focus on confusing classes only. (3) However, it hurts the performance when combining with other distilation approaches, such as Sparsemax+ME and Sparsemax+PL, becuase they fails to target on the most informative predictions. Finally we conclude that sparsemax and Sharpening both contributes to ADS.

G. Scarce Labeled Data Scenario
One of the challenges in SSL is that in some scenarios only a limited number of labeled data are provided. FixMatch has shown its superiority to deal with the scarce labeled data thanks to its tailor-designed consistency loss. To confirm that the proposed ADS also applies to this case, we conduct FixMatch-ADS with a small fraction of labeled data. Three datasets CIFAR-10, CIFAR-100, and SVHN with all but 40, 400, 40 labeled data are used following the setting of [16], and the results are shown in Table IV. We observe that ADS achieves better performance than the naive FixMatch across all the three experiment settings. In particular, the improvement on SVHN is significant (more than 5%), which demonstrates the efficacy of ADS in the scenario with scare labeled data.

VII. DISCUSSION
In this paper, we have presented a new prediction distillation mechanism named ADaptive Sharpening (ADS) which distills example-wise informed predictions to avert the risk of overconfident predictions. We discriminate ADS from existing distillation strategies from the perspective of prediction uncertainty minimization. By integrating ADS into SSL framework we discuss how ADS affects other SSL components. The theoretical analyses present the valuable properties of ADS which confirm our motivation. Extensive experiments demonstrate the proposed ADS forges a cornerstone for distillation-based SSL research. In future work, we would apply ADS to more SSL tasks including different types of data [36], [37] and unseen class scenarios [38].
where e is Euler's number. Note that Eq. (20) is a binary case for Theorem 1.
Considering the distillation loss with the power r = 2, we have the target probability where t is calculated as follows.
It is easily verified that t(s) is an odd function and t | s=0.5 = 0.5. Based on Eq. (21), we know the distillation loss (Eq. (10)) is nonzero iff 1 e+1 ≤ s ≤ e e+1 , As t will stop the gradients from propagation, let s := (ln s 1−s + 1)/2, and only the reduced form has the gradient The above derivatives are used to complete the plot of ADS in Figure 3, and other distillation methods are displayed in a similar manner.
APPENDIX B PROOF FOR COROLLARY 1 Corollary 1. For a K-way semi-supervised classification problem, the determinate predictions and negligible predictions for ADS are masked out by the sample dependent threshold θ 1 ∈ [ e e+K−1 , e e+1 ] and θ 2 ∈ [ e ρ ρ+e ρ (K−ρ) , e ρ ρ+e ρ ] in the corresponding softmax output space, respectively, where e is Euler number and ρ is the population of non-zero predictions.