summaries: Proposes cycle self-training (CST) an unsupervised domain adaptation (UDA) algorithm that cycles between standard self-training of a target classifier on (source model generated) target pseudolabels, and a new cycle self-training step that updates backbone representations so as to generalize the target classifier to the source domain. An additional self-training objective based on the Tsallis entropy is introduced. Theoretical and empirical results are presented on several standard UDA benchmarks. The paper tackles the problem of domain adaptation with a newly proposed cycle self-training algorithm. Given observed that the pseudo labels are noisy and existing de-noising methods require ad-hoc hyper-parameters for specific tasks, the authors attempt to progressively refine the pseudo labels by the capability of the network itself. With the intuition of transferring the source-domain knowledge to the target domain, the authors propose to regularize the pseudo labels with a cyclic training pipeline. Tsallis Entropy is further introduced to improve the label quality. Experiments on both visual classification and linguistic sentiment classification indicate the effectiveness of the introduced method. This paper presents a novel approach for unsupervised domain adaptation (UDA) based on cycle self-training. Instead of conventional self-training, there are a generated and two attached classifiers with different purposes. The source classifier, mainly trained on the source loss, is applied to infer pseudo target labels. The target classifier is trained by the self-training loss based on the pseudo labels provided by the source classifier.  The authors also propose the Tsallis entropy as the regularization term, which can be seen as a side contribution. This paper proposes an unsupervised domain adaptation method based on cyclical self-training. There are two main components to the method: 1. a cyclical self-training algorithm that solves a bilevel optimization problem based on an in the inner loop which trains a target classifier with target pseudo-labels  and an outer loop that makes the target classifier perform well on the source domain by updating the shared representations in the outer loop. 2. an uncertainty measure based on the Tsallis entropy, which adaptively minimizes the uncertainty in target pseudo-labels, which replaces the standard Gibbs entropy used by self-training methods. Experiments on computer vision and NLP datasets show that proposed method outperforms baseline methods by a decent margin. Ablation studies show that both components of the algorithm---cyclical self-training and Tsallis entropy---meaningfully contribute to performance improvement. Qualitative and quantitative studies show that target pseudo-label quality is improved by the proposed approach. 
abstractText: Mainstream approaches for unsupervised domain adaptation (UDA) learn domaininvariant representations to narrow the domain shift, which are empirically effective but theoretically challenged by the hardness or impossibility theorems. Recently, self-training has been gaining momentum in UDA, which exploits unlabeled target data by training with target pseudo-labels. However, as corroborated in this work, under distributional shift, the pseudo-labels can be unreliable in terms of their large discrepancy from target ground truth. In this paper, we propose Cycle Self-Training (CST), a principled self-training algorithm that explicitly enforces pseudo-labels to generalize across domains. CST cycles between a forward step and a reverse step until convergence. In the forward step, CST generates target pseudo-labels with a source-trained classifier. In the reverse step, CST trains a target classifier using target pseudo-labels, and then updates the shared representations to make the target classifier perform well on the source data. We introduce the Tsallis entropy as a confidence-friendly regularization to improve the quality of target pseudo-labels. We analyze CST theoretically under realistic assumptions, and provide hard cases where CST recovers target ground truth, while both invariant feature learning and vanilla self-training fail. Empirical results indicate that CST significantly improves over the state-of-the-arts on visual recognition and sentiment analysis benchmarks.
authors.name: Hong Liu
authors.name: Jianmin Wang
references.authors: E.A. Albadawy A. Saha M.A. Mazurowski
references.title: Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing
references.venue: Medical Physics,
references.authors: E. Arazo D. Ortego P. Albert N.E. O’Connor K. McGuinness
references.title: Pseudo-labeling and confirmation bias in deep semi-supervised learning
references.authors: S. Arora S.S. Du W. Hu Z. Li R.R. Salakhutdinov R. Wang
references.title: On exact computation with an infinitely wide neural net
references.venue: In NeurIPS,
references.authors: P. Bachman O. Alsharif D. Precup
references.title: Learning with pseudo-ensembles
references.venue: In NeurIPS,
references.authors: P.L. Bartlett S. Mendelson
references.title: Rademacher and gaussian complexities: Risk bounds and structural results
references.venue: JMLR, 3(Nov):463–482,
references.authors: S. Ben-David R. Urner
references.title: On the hardness of domain adaptation and the utility of unlabeled target samples
references.venue: In ALT,
references.authors: S. Ben-David J. Blitzer K. Crammer A. Kulesza F. Pereira J.W. Vaughan
references.title: A theory of learning from different domains
references.venue: Machine Learning,
references.authors: D. Berthelot N. Carlini I. Goodfellow N. Papernot A. Oliver C. Raffel
references.title: Mixmatch: A holistic approach to semi-supervised learning
references.authors: L. Bertinetto J.F. Henriques P. Torr A. Vedaldi
references.title: Meta-learning with differentiable closed-form solvers
references.venue: In ICLR,
references.authors: J. Blitzer M. Dredze F. Pereira
references.title: Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification
references.venue: In ACL,
references.authors: T. Cai R. Gao J.D. Lee Q. Lei
references.title: A theory of label propagation for subpopulation
references.authors: N. Carlini
references.title: Poisoning the unlabeled dataset of semi-supervised learning, 2021
references.authors: C. Chen W. Xie W. Huang Y. Rong X. Ding Y. Huang T. Xu J. Huang
references.title: Progressive feature alignment for unsupervised domain adaptation
references.venue: In CVPR,
references.authors: Y. Chen C. Wei A. Kumar T. Ma
references.title: Self-training avoids using spurious features under domain shift
references.venue: In NeurIPS,
references.authors: J. Devlin Chang M.-W K. Lee K. Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL,
references.authors: J. Du E. Grave B. Gunel V. Chaudhary O. Celebi M. Auli V. Stoyanov A. Conneau
references.title: Self-training improves pre-training for natural language understanding
references.venue: In NAACL,
references.authors: C. Finn P. Abbeel S. Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In ICML,
references.authors: P. Foret A. Kleiner H. Mobahi B. Neyshabur
references.title: Sharpness-aware minimization for efficiently improving generalization
references.venue: In ICLR,
references.authors: S. Frei D. Zou Z. Chen Q. Gu
references.title: Self-training converts weak learners to strong learners in mixture models
references.venue: arXiv preprint arXiv:2106.13805,
references.authors: G. French M. Mackiewicz M. Fisher
references.title: Self-ensembling for visual domain adaptation
references.venue: In ICLR,
references.authors: Y. Ganin E. Ustinova H. Ajakan P. Germain H. Larochelle M. Marchand V. Lempitsky
references.title: Domain-adversarial training of neural networks
references.authors: G. Ghiasi B. Zoph E.D. Cubuk Q.V. Le Lin T.-Y
references.title: Multi-task self-training for learning general representations
references.venue: In ICCV,
references.authors: Y. Grandvalet Y. Bengio
references.title: Semi-supervised learning by entropy minimization
references.venue: In NeurIPS, pp
references.authors: C. Guo G. Pleiss Y. Sun K.Q. Weinberger
references.title: On calibration of modern neural networks
references.venue: In ICML,
references.authors: K. He X. Zhang S. Ren J. Sun
references.title: Deep residual learning for image recognition
references.venue: In CVPR,
references.authors: J. Hoffman E. Tzeng T. Park J. Zhu P. Isola K. Saenko A.A. Efros T. Darrell
references.title: Cycada: Cycle-consistent adversarial domain adaptation
references.venue: In ICML,
references.authors: X. Jiang Q. Lao S. Matwin M. Havaei
references.title: Implicit class-conditioned domain alignment for unsupervised domain adaptation
references.venue: In ICML,
references.authors: F.D. Johansson D. Sontag R. Ranganath
references.title: Support and invertibility in domain-invariant representations
references.venue: In AISTATS,
references.authors: A. Kumar T. Ma P. Liang
references.title: Understanding self-training for gradual domain adaptation
references.venue: In ICML,
references.authors: Lee D.-H
references.title: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop: Challenges in Representation Learning (WREPL)
references.authors: B. Li Y. Wang T. Che S. Zhang S. Zhao P. Xu W. Zhou Y. Bengio K. Keutzer
references.title: Rethinking distributional matching based domain adaptation
references.authors: H. Liu M. Long J. Wang M. Jordan
references.title: Transferable adversarial training: A general approach to adapting deep classifiers
references.venue: In ICML,
references.authors: M. Long Y. Cao J. Wang M.I. Jordan
references.title: Learning transferable features with deep adaptation networks
references.venue: In ICML, pp
references.authors: M. Long H. Zhu J. Wang M.I. Jordan
references.title: Unsupervised domain adaptation with residual transfer networks
references.venue: In NeurIPS,
references.authors: M. Long H. Zhu J. Wang M.I. Jordan
references.title: Deep transfer learning with joint adaptation networks
references.venue: In ICML,
references.authors: M. Long Z. Cao J. Wang M.I. Jordan
references.title: Conditional adversarial domain adaptation
references.venue: In NeurIPS,
references.authors: Z. Lu Y. Yang X. Zhu C. Liu Song Y.-Z T. Xiang
references.title: Stochastic classifiers for unsupervised domain adaptation
references.venue: In CVPR,
references.authors: A. Mey M. Loog
references.title: A soft-labeled self-training approach
references.venue: In ICPR,
references.authors: T. Miyato S. Maeda S. Ishii M. Koyama
references.title: Virtual adversarial training: A regularization method for supervised and semi-supervised learning
references.authors: S. Mukherjee A. Awadallah
references.title: Uncertainty-aware self-training for few-shot text classification
references.venue: In NeurIPS,
references.authors: A. Paszke S. Gross F. Massa A. Lerer J. Bradbury G. Chanan T. Killeen Z. Lin N. Gimelshein L. Antiga A. Desmaison A. Kopf E. Yang Z. DeVito M. Raison A. Tejani S. Chilamkurthy B. Steiner L. Fang J. Bai S. Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: In NeurIPS,
references.authors: X. Peng B. Usman N. Kaushik J. Hoffman D. Wang K. Saenko
references.title: Visda: The visual domain adaptation challenge
references.authors: X. Peng Q. Bai X. Xia Z. Huang K. Saenko B. Wang
references.title: Moment matching for multi-source domain adaptation
references.venue: In ICCV,
references.authors: V. Prabhu S. Khare D. Kartik J. Hoffman
references.title: Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation, 2020
references.authors: V. Prabhu S. Khare D. Kartik J. Hoffman
references.title: Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation
references.venue: In ICCV,
references.authors: X. Qu Z. Zou Y. Cheng Y. Yang P. Zhou
references.title: Adversarial category alignment network for cross-domain sentiment classification
references.authors: J. Quionero-Candela M. Sugiyama A. Schwaighofer N.D. Lawrence
references.title: Dataset Shift in Machine Learning
references.authors: A. Rasmus M. Berglund M. Honkala H. Valpola T. Raiko
references.title: Semi-supervised learning with ladder networks
references.venue: In NeurIPS,
references.authors: C. Rosenberg M. Hebert H. Schneiderman
references.title: Semi-supervised self-training of object detection models
references.venue: In WACV,
references.authors: K. Saito K. Watanabe Y. Ushiku T. Harada
references.title: Maximum classifier discrepancy for unsupervised domain adaptation
references.venue: In CVPR,
references.authors: M. Sajjadi M. Javanmardi T. Tasdizen
references.title: Regularization with stochastic transformations and perturbations for deep semi-supervised learning
references.venue: In NeurIPS,
references.authors: R. Shu H. Bui H. Narui S. Ermon
references.title: A DIRT-t approach to unsupervised domain adaptation
references.venue: In ICLR,
references.authors: K. Sohn D. Berthelot N. Carlini Z. Zhang H. Zhang C.A. Raffel E.D. Cubuk A. Kurakin Li C.-L
references.title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence
references.venue: NeurIPS,
references.authors: C. Szegedy W. Zaremba I. Sutskever J. Bruna D. Erhan I. Goodfellow R. Fergus
references.title: Intriguing properties of neural networks
references.venue: In ICLR,
references.authors: M. Talagrand
references.title: Upper and lower bounds for stochastic processes: modern methods and classical problems, volume 60
references.venue: Springer Science & Business Media,
references.authors: S. Tan X. Peng K. Saenko
references.title: Class-imbalanced domain adaptation: An empirical odyssey
references.venue: In ECCV Workshop,
references.authors: A. Tarvainen H. Valpola
references.title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
references.venue: In NeurIPS,
references.authors: C. Tsallis
references.title: Possible generalization of boltzmann-gibbs statistics
references.venue: Journal of Statistical Physics,
references.authors: E. Tzeng J. Hoffman K. Saenko T. Darrell
references.title: Adversarial discriminative domain adaptation
references.venue: In CVPR,
references.authors: H. Venkateswara J. Eusebio S. Chakraborty S. Panchanathan
references.title: Deep hashing network for unsupervised domain adaptation
references.venue: In CVPR,
references.authors: T. Vu Luong M.-T Q.V. Le G. Simon M. Iyyer
references.title: Strata: Self-training with task augmentation for better few-shot learning
references.venue: arXiv preprint arXiv:2109.06270,
references.authors: C. Wei K. Shen C. Yining T. Ma
references.title: Theoretical analysis of self-training with deep networks on unlabeled data
references.venue: In ICLR,
references.authors: Q. Xie M.T. Luong E. Hovy Q.V. Le
references.title: Self-training with noisy student improves imagenet classification
references.venue: In CVPR,
references.authors: S.M. Xie A. Kumar R. Jones F. Khani T. Ma P. Liang
references.title: In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness
references.venue: In ICLR,
references.authors: R. Xu G. Li J. Yang L. Lin
references.title: Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation
references.authors: J. Yosinski J. Clune Y. Bengio H. Lipson
references.title: How transferable are features in deep neural networks
references.venue: In NeurIPS,
references.authors: W. Zellinger T. Grubinger E. Lughofer T. Natschläger S. Saminger-Platz
references.title: Central moment discrepancy (CMD) for domain-invariant representation learning
references.venue: In ICLR,
references.authors: H. Zhang M. Cisse Y.N. Dauphin D. Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: In ICLR,
references.authors: Y. Zhang T. Liu M. Long M. Jordan
references.title: Bridging theory and algorithm for domain adaptation
references.venue: In ICML,
references.authors: H. Zhao R.T.D. Combes K. Zhang G. Gordon
references.title: On learning invariant representations for domain adaptation
references.venue: In ICML,
references.authors: Y. Ziser R. Reichart
references.title: Pivot based language modeling for improved neural domain adaptation
references.venue: In NAACL,
references.authors: B. Zoph G. Ghiasi Lin T.-Y Y. Cui H. Liu E.D. Cubuk Q. Le
references.title: Rethinking pre-training and self-training
references.venue: In NeurIPS,
references.authors: Y. Zou Z. Yu B.V.K. Vijaya Kumar J. Wang
references.title: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training
references.venue: In ECCV,
references.authors: Y. Zou Z. Yu X. Liu B.V. Kumar J. Wang
references.title: Confidence regularized self-training
references.venue: In ICCV, October 2019
sections.heading: 1 Introduction
sections.text: Transferring knowledge from a source domain with rich supervision to an unlabeled target domain is an important yet challenging problem. Since deep neural networks are known to be sensitive to subtle change in underlying distributions [70], models trained on one labeled dataset often fail to generalize to another unlabeled dataset [58, 1]. Unsupervised domain adaptation (UDA) addresses the challenge of distributional shift by adapting the source model to the unlabeled target data [50, 43].
  The mainstream paradigm for UDA is feature adaptation, a.k.a. domain alignment. By reducing the distance of the source and target feature distributions, these methods learn invariant representations to facilitate knowledge transfer between domains [34, 22, 36, 54, 37, 73], with successful applications in various areas such as computer vision [63, 27, 77] and natural language processing [75, 49]. Despite their popularity, the impossibility theories [6] uncovered intrinsic limitations of learning invariant representations when it comes to label shift [74, 32] and shift in the support of domains [29].
  Recently, self-training (a.k.a. pseudo-labeling) [21, 78, 30, 32, 47, 68] has been gaining momentum as a promising alternative to feature adaptation. Originally tailored to semi-supervised learning, self-training generates pseudo-labels of unlabeled data, and jointly trains the model with source labels and target pseudo-labels [31, 39, 30]. However, the distributional shift in UDA makes pseudo-labeling more difficult. Directly using all pseudo-labels is risky due to accumulated error and even trivial solution [14]. Thus previous works tailor self-training to UDA by selecting trustworthy pseudo-labels. Using confidence threshold or reweighting, recent works try to alleviate the negative effect of domain
  ⇤Corresponding author: Mingsheng Long (mingsheng@tsinghua.edu.cn)
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  shift in standard self-training [78, 47], but they can be brittle and require expensive tweaking of the threshold or weight for different tasks, and their performance gain is still inconsistent.
  In this work, we first analyze the quality of pseudo-labels with or without domain shift to delve deeper into the difficulty of standard self-training in UDA. On popular benchmark datasets, when the source and target are the same, our analysis indicates that the pseudo-label distribution is almost identical to the ground-truth distribution. However, with distributional shift, their discrepancy can be very large with examples of several classes mostly misclassified into other classes. We also study the difficulty of selecting correct pseudo-labels with popular criteria under domain shift. Although entropy and confidence are reasonable selection criteria for correct pseudo-labels without domain shift, the domain shift makes their accuracy decrease sharply.
  Our analysis shows that domain shift makes pseudo-labels unreliable and that self-training on selected target instances with accurate pseudo-labels is less successful. Thereby, more principled improvement of standard self-training should be tailored to UDA and address the domain shift explicitly. In this work, we propose Cycle Self-Training (CST), a principled self-training approach to UDA, which overcomes the limitations of standard self-training (see Figure 1). Different from previous works to select target pseudo-labels with hard-to-tweak protocols, CST learns to generalize the pseudo-labels across domains. Specifically, CST cycles between the use of target pseudo-labels to train a target classifier, and the update of shared representations to make the target classifier perform well on the source data. In contrast to the standard Gibbs entropy that makes the target predictions over-confident, we propose a confidence-friendly uncertainty measure based on the Tsallis entropy in information theory, which adaptively minimizes the uncertainty without manually tuning or setting thresholds. Our method is simple and generally applicable to vision and language tasks with various backbones.
  We empirically evaluate our method on a series of standard UDA benchmarks. Results indicate that CST outperforms previous state-of-the-art methods in 21 out of 25 tasks for object recognition and sentiment classification. Theoretically, we prove that the minimizer of CST objective is endowed with general guarantees of target performance. We also study hard cases on specific distributions, showing that CST recovers target ground-truths while both feature adaptation and standard self-training fail.
sections.heading: 2 Preliminaries
sections.text: We study unsupervised domain adaptation (UDA). Consider a source distribution P and a target distribution Q over the input-label space X ⇥ Y . We have access to ns labeled i.i.d. samples bP = {xsi , ysi } ns i=1 from P and nt unlabeled i.i.d. samples bQ = {xti} nt i=1 from Q. The model f comprises a feature extractor h parametrized by and a head (linear classifier) g✓ parametrized by ✓, i.e. f✓, (x) = g✓(h (x)). The loss function is `(·, ·). Denote by LP (✓, ) := E(x,y)⇠P `(f✓, (x), y) the expected error on P . Similarly, we use L bP (✓, ) to denote the empirical error on dataset bP . We discuss two mainstream UDA methods and their formulations: feature adaptation and self-training.
  Feature Adaptation trains the model f on the source dataset bP , and simultaneously matches the source and target distributions in the representation space Z = h(X ):
  min ✓,
  L bP (✓, ) + d(h] bP , h] bQ). (1)
  Here, h] bP denotes the pushforward distribution of bP , and d(·, ·) is some distribution distance. For instance, Long et al. [34] used maximum mean discrepancy dMMD, and Ganin et al. [22] approximated the H H-distance dH H [7] with adversarial training. Despite its pervasiveness, recent works have shown the intrinsic limitations of feature adaptation under real-world situations [6, 74, 33, 32, 29].
  Self-Training is considered a promising alternative to feature adaptation. In this work we mainly focus on pseudo-labeling [31, 30]. Stemming from semi-supervised learning, standard self-training trains a source model fs on the source dataset bP : min✓s, s L bP (✓s, s). The target pseudo-labels are then generated by fs on the target dataset bQ. To leverage unlabeled target data, self-training trains the model on the source and target datasets together with source ground-truths and target pseudo-labels:
  min ✓, L bP (✓, ) + Ex⇠ bQ`(f✓, (x), argmax i {f✓s, s(x)[i]}). (2)
  Self-training also uses label-sharpening as a standard protocol [31, 57]. Another popular variant of pseudo-labeling is the teacher-student model [4, 61], which iteratively improves the quality of pseudo-labels via alternatively replacing ✓s and s with ✓ and of the previous iteration.
sections.heading: 2.1 Limitations of Standard Self-Training
sections.text: Standard self-training with pseudo-labels uses unlabeled data efficiently for semi-supervised learning [31, 39, 57]. Here we carry out exploratory studies on the popular VisDA-2017 [45] dataset using ResNet-50 backbones. We find that domain shift makes the pseudo-labels biased towards several classes and thereby unreliable in UDA. See Appendix C.1 for details and results on more datasets.
  Pseudo-label distributions with or without domain shift. We resample the original VisDA-2017 to simulate different relationship between source and target domains: 1) i.i.d., 2) covariate shift, and 3) label shift. We train the model on the three variants of source dataset and use it to generate target pseudo-labels. We show the distributions of target ground-truths and pseudo-labels in Figure 2 (Left). When the source and target distributions are identical, the distribution of pseudo-labels is almost the same as ground-truths, indicating the reliability of pseudo-labels. In contrast, when exposed to label shift or covariate shift, the distribution of pseudo-labels is significantly different from target ground-truths. Note that classes 2, 7, 8 and 12 appear rarely in the target pseudo-labels in the covariate shift setting, indicating that the pseudo-labels are biased towards several classes due to domain shift. Self-training with these pseudo-labels is risky since it may lead to misalignment of distributions and misclassify many examples of classes 2, 7, 8 and 12.
  Change of pseudo-label distributions throughout training. To further study the change of pseudolabels in standard self-training, we compute the total variation (TV) distance between target groundtruths and target pseudo-labels: dTV(c, c0) = 12 P i kci c 0
  ik, where ci is the ratio of class i. We plot its change during training in Figure 2 (Middle). Although the error rate of pseudo-labels continues to decrease, dTV remains almost unchanged at 0.26 throughout training. Note that dTV is the lower bound of the error rate of the pseudo-labels (shown in Appendix C.1). If dTV converges to 0.26, then the accuracy of pseudo-labels is upper-bounded by 0.74. This indicates that the important denoising ability [66] of pseudo-labels in standard self-training is hindered by domain shift.
  Difficulty of selecting reliable pseudo-labels under domain shift. To mitigate the negative effect of false pseudo-labels, recent works proposed to select correct pseudo-labels based on thresholding the entropy or confidence criteria [35, 21, 37, 57]. However, it remains unclear whether these strategies are still effective under domain shift. Here we compare the quality of pseudo-labels selected by
  different strategies with or without domain shift. For each strategy, we compute False Positive Rate and True Positive Rate for different thresholds and plot its ROC curve in Figure 2 (Right). When the source and target distributions are identical, both entropy and confidence are reasonable strategies for selecting correct pseudo-labels (AUC=0.89). However, when the target pseudo-labels are generated by the source model, the quality of pseudo-labels decreases sharply under domain shift (AUC=0.78).
sections.heading: 3 Approach
sections.text: We present Cycle Self-Training (CST) to improve pseudo-labels under domain shift. An overview of our method is given in Figure 1. Cycle Self-Training iterates between a forward step and a reverse step to make self-trained classifiers generalize well on both target and source domains.
sections.heading: 3.1 Cycle Self-Training
sections.text: Forward Step. Similar to standard self-training, we have a source classifier ✓s trained on top of the shared representations on the labeled source domain, and use it to generate target pseudo-labels as
  y 0 = argmax i {f✓s, (x)[i]}, (3)
  for each x in the target dataset bQ. Traditional self-training methods use confidence thresholding or reweighting to select reliable pseudo-labels. For example, Sohn et al. [57] select pseudo-labels with softmax value and Long et al. [37] add entropy reweighting to rely on examples with more confidence prediction. However, the output of deep networks is usually miscalibrated [25], and is not necessarily related to the ground-truth confidence even on the same distribution. In domain adaptation, as shown in Section 2.1, the discrepancy between the source and target domains makes pseudo-labels even more unreliable, and the performance of commonly used selection strategies is also unsatisfactory. Another drawback is the expensive tweaking in order to find the optimal confidence threshold for new tasks. To better apply self-training to domain adaptation, we expect that the model can gradually refine the pseudo-labels by itself without the cumbersome selection or thresholding.
  Reverse Step. We design a complementary step with the following insights to improve self-training. Intuitively, the labels on the source domain contain both useful information that can transfer to the target domain and harmful information that can make pseudo-labels incorrect. Similarly, reliable pseudo-labels on the target domain can transfer to the source domain in turn, while models trained with incorrect pseudo-labels on the target domain cannot transfer to the source domain. In this sense, if we explicitly train the model to make target pseudo-labels informative of the source domain, we can gradually make the pseudo-labels more accurate and learn to generalize to the target domain.
  Specifically, with the pseudo-labels y0 generated by the source classifier ✓s at hand as in equation 3, we train a target head ✓̂t( ) on top of the representation with pseudo-labels on the target domain bQ,
  ✓̂t( ) = argmin ✓ Ex⇠ bQ`(f✓, (x), y 0). (4)
  We wish to make the target pseudo-labels informative of the source domain and gradually refine them. To this end, we update the shared feature extractor to predict accurately on the source domain and jointly enforce the target classifier ✓̂t( ) to perform well on the source domain. This naturally leads to the objective of Cycle Self-Training:
  minimize ✓s, LCycle(✓s, ) := L bP (✓s, ) + L bP (✓̂t( ), ). (5)
  Bi-level Optimization. The objective in equation 5 relies on the solution ✓̂t( ) to the objective in equation 4. Thus, CST formulates a bi-level optimization problem. In the inner loop we generate target pseudo-labels with the source classifier (equation 3), and train a target classifier with target pseudo-labels (equation 4). After each inner loop, we update the feature extractor for one step in the outer loop (equation 5), and start a new inner loop again. However, since the inner loop of the optimization in equation 4 only involves the light-weight linear head ✓t, we propose to calculate the analytical form of ✓̂t( ) and directly back-propagate to the feature extractor instead of calculating the second-order derivatives as in MAML [18]. The resulting framework is as fast as training two heads jointly. Also note that the solution ✓̂t( ) relies on ✓s implicitly through y0. However, both standard self-training and our implementation use label sharpening, making y0 not differentiable. Thus we follow vanilla self-training and do not consider the gradient of ✓̂t( ) w.r.t. y0 in the outer loop optimization. We defer the derivation and implementation of bi-level optimization to Appendix B.2.
sections.heading: 3.2 Tsallis Entropy Minimization
sections.text: Gibbs entropy is widely used by existing semi-supervised learning methods to regularize the model output and minimize the uncertainty of predictions on unlabeled data [24]. In this work, we generalize Gibbs entropy to Tsallis entropy [62] in information theory. Suppose the softmax output of a model is y 2 RK , then the ↵-Tsallis entropy is defined as
  S↵(y) = 1
  ↵ 1
  ⇣ 1 X y ↵ [i] ⌘ , (6)
  where ↵ > 0 is the entropic-index. Note that lim↵!1 S↵(y) = P
  i y[i]log(y[i]) which exactly recovers the Gibbs entropy. When ↵ = 2, S↵(y) becomes the Gini impurity 1 P i y 2 [i].
  We propose to control the uncertainty of target pseudo-labels based on Tsallis entropy minimization: L bQ,Tsallis,↵(✓, ) := Ex⇠ bQS↵(f✓, (x)). (9)
  Figure 3 shows the change of Tsallis entropy with different entropic-indices ↵ for binary problems. Intuitively, smaller ↵ exerts more penalization on uncertain predictions and larger ↵ allows several scores yi’s to be similar. This is critical in self-training since an overly small ↵ (as in Gibbs entropy) will make the incorrect dimension of pseudo-labels close to 1 and have no chance to be corrected throughout training. In Section 5.4, we further verify this property with experiments.
  An important improvement of the Tsallis entropy over Gibbs entropy is that it can choose the suitable measure of uncertainty for different systems to avoid over-confidence caused by overly penalizing the uncertain pseudo-labels. To automatically find the suitable ↵, we adopt a similar strategy as Section 3.1. The intuition is that if we use the suitable entropic-index ↵ to train the source classifier ✓s,↵, the target pseudo-labels generated by ✓s,↵ will contain desirable knowledge of the source dataset, i.e. a target classifier ✓t,↵ trained with these pseudo-labels will perform well on the source domain. Therefore, we semi-supervisedly train a classifier ✓̂s,↵ on the source domain with the ↵-Tsallis entropy regularization L bQ,Tsallis,↵ on the target domain as: ✓̂s,↵ = argmin✓ L bP (✓, )+L bQ,Tsallis,↵(✓, ), from which we obtain the target pseudo-labels. Then we train another head ✓̂t,↵ with target pseudo-labels. We automatically find ↵ by minimizing the loss of ✓̂t,↵ on the source data:
  ↵̂ = argmin ↵2[1,2] L bP (✓̂t,↵, ) (10)
  To solve equation 10, we discretize the feasible region [1, 2] of ↵ and use discrete optimization to lower computational cost. We also update ↵ at the start of each epoch, since we found more frequent
  Algorithm 1 Cycle Self-Training (CST) 1: Input: source dataset bP and target dataset bQ. 2: for epoch = 0 to MaxEpoch do 3: Select ↵̂ as equation 10 at the start of each epoch. 4: for t = 0 to MaxIter do 5: Forward Step 6: Generate pseudo-labels on the target domain with and ✓s: y0 = argmaxi{f✓s, (x)[i]}. 7: Reverse Step 8: Train a target head ✓̂t( ) with target pseudo-labels y0 on the feature extractor :
  ✓̂t( ) = argmin ✓ Ex⇠ bQ`(f✓, (x), y 0).
  9: Update the feature extractor and the source head ✓s to make ✓̂t( ) perform well on the source dataset and minimize the ↵̂-Tsallis entropy on the target dataset:
  ⌘r [L bP (✓s, ) + L bP (✓̂t( ), ) + L bQ,Tsallis,↵̂(✓s, )]. (7)
  ✓s ✓s ⌘r✓s [L bP (✓s, ) + L bQ,Tsallis,↵̂(✓s, )]. (8)10: end for 11: end for
  update leads to no performance gain. Details are deferred to Appendix B.3. Finally, with the optimal ↵̂ found, we add the ↵̂-Tsallis entropy minimization term L bQ,Tsallis,↵̂ to the overall objective:
  minimize ✓s, LCycle(✓s, ) + L bQ,Tsallis,↵̂(✓s, ). (11)
  In summary, Algorithm 1 depicts the complete training procedure of Cycle Self-Training (CST).
sections.heading: 4 Theoretical Analysis
sections.text: We analyze the properties of CST theoretically. First, we prove that the minimizer of the CST loss LCST(fs, ft) will lead to small target loss ErrQ(fs) under a simple but realistic expansion assumption. Then, we further demonstrate a concrete instantiation where cycle self-training provably recovers the target ground truth, but both feature adaptation and standard self-training fail. Due to space limit, we state the main results here and defer all proof details to Appendix A.
sections.heading: 4.1 CST Provably Works under the Expansion Assumption
sections.text: We start from a K-way classification model, f : X ! [0, 1]K 2 F and f̃(x) := argmaxi f(x)[i] denotes the prediction. Denote by Pi the conditional distribution of P given y = i. Assume the supports of Pi and Pj are disjoint for i 6= j. The definition is similar for Qi. We further Assume P (y = i) = Q(y = i). For any x 2 X , N (x) is defined as the neighboring set of x with a proper metric d(·, ·), N (x) = {x0 : d(x, x0)  ⇠}. N (A) := [x2AN (x). Denote the expected error on the target domain by ErrQ(f) := E(x,y)⇠QI(f̃(x) 6= y). We study the CST algorithm under the expansion assumption of the mixture distribution [66, 11]. Intuitively, this assumption indicates that the conditional distributions Pi and Qi are closely located and regularly shaped, enabling knowledge transfer from the source domain to the target domain. Definition 1 ((q, ✏)-constant expansion [66]). We say P and Q satisfy (q, ✏)-constant expansion for some constant q, ✏ 2 (0, 1), if for any set A 2 X and any i 2 [K] with 12 > P 12 (Pi+Qi)(A) > q, we have P 1
  2 (Pi+Qi) (N (A)\A) > min{✏, P 1 2 (Pi+Qi) (A)}.
  Based on this expansion assumption, we consider a robustness-constrained version of CST. Later we will show that the robustness is closely related to the uncertainty. Denote by fs the source model and ft the model trained on the target with pseudo-labels. Let R(ft) := P 1
  2 (P+Q) ({x : 9x0 2
  N (x), f̃t(x) 6= f̃t(x0)}) represent the robustness [66] of ft on P and Q. Suppose E(x,y)⇠QI(f̃s(x) 6= f̃t(x))  c and R(ft)  ⇢. The following theorem states that when fs and ft behave similarly on the target domain Q and ft is robust to local changes in input, the minimizer of the cycle source error ErrP (ft) will guarantee low error of fs on the target domain Q.
  Theorem 1. Suppose Definition 1 holds for P and Q. For any fs, ft satisfying E(x,y)⇠QI(f̃s(x) 6= f̃t(x))  c and R(ft)  ⇢, the expected error of fs on the target domain Q is bounded,
  ErrQ(fs)  ErrP (ft) + c+ 2q + ⇢
  min{✏, q} . (12)
  To further relate the expected error with the CST training objective and obtain finite-sample guarantee, we use the multi-class margin loss: l (f(x), y) := ( M(f(x), y)), where M(v, y) = v[y] maxy0 6=y v[y0] and is the ramp function. We then extend the margin loss: M(v) = maxy(v[y] maxy0 6=y v[y0]) (The difference between the largest and the second largest scores in v), and l (ft(x), fs(x)) := ( M(ft(x), f̃s(x))). Further suppose f[i] is Lf -Lipschitz w.r.t. the metric d(·, ·) and ⌧ := 1 2Lf⇠min{✏, q} > 0. Consider the following training objective for CST, denoted by LCST(fs, ft), where L bP, (ft) := E(x,y)⇠ bP l (ft(x), y) corresponds to the cycle source loss in equation 5, L bQ, (ft, fs) := E(x,y)⇠ bQl (ft(x), fs(x)) is consistent with the target loss in equation 4, and M(ft(x)) is closely related to the uncertainty of predictions in equation 11.
  minLCST(fs, ft) := L bP, (ft) + L bQ, (ft, fs) + 1 E(x,y)⇠ 12 ( bP+ bQ)M(ft(x))
  ⌧ . (13)
  The following theorem shows that the minimizer of the training objective LCST(fs, ft) guarantees low population error of fs on the target domain Q.
  Theorem 2. bR(F| bP ) denotes the empirical Rademacher complexity of function class F on dataset bP . For any solution of equation 13 and > 0, with probability larger than 1 ,
  ErrQ(fs)  LCST(fs, ft) + 2q + 4K h bR(F| bP ) + bR(F̃ ⇥ F| bQ) i + 2
  ⌧
  h bR(F| bP ) + bR(F| bQ) i + ⇣,
  where ⇣ = O ⇣p log(1/ )/ns + p log(1/ )/nt ⌘
  is a low-order term. F̃ ⇥ F refers to the function class {x! f(x)[f̃ 0(x)] : f, f 0 2 F}.
  Main insights. Theorem 2 justifies CST under the expansion assumption. The generalization error of the classifier fs on the target domain is bounded with the CST loss objective LCST(fs, ft), the intrinsic property of the data distribution q, and the complexity of the function classes. In our algorithm, LCST(fs, ft) is minimized by the neural networks and q is a constant. The complexity of the function class can be controlled with proper regularization.
sections.heading: 4.2 Hard Case for Feature Adaptation and Standard Self-Training
sections.text: To gain more insight, we study UDA in a quadratic neural network f✓, (x) = ✓>( >x) 2, where is element-wise power. In UDA, the source can have multiple solutions but we aim to learn the one working on the target [34]. We design the underlying distributions p and q in Table 6 to reflect this. Consider the following P and Q. x[1] and x[2] are sampled i.i.d. from distribution p on P , and from
  Table 1: The design of p and q.
  Distribution 1 +1 0 Source p 0.05 0.05 0.90 Target q 0.25 0.25 0.50
  q on Q. For i 2 [3, d], x[i] = ix[2] on P and x[i] = ix[1] on Q. i 2 {±1} are i.i.d. and uniform. We also assume realizability: y = x2[1] x 2 [2] for both source and target. Note that y = x 2 [1] x 2 [i]
  for all i 2 [2, d] are solutions to P but only y = x2[1] x 2 [2] works on Q. We visualize this specialized setting in Figure 4.
  To make the features more tractable, we study the norm-constrained version of the algorithms (details are deferred to Section A.3.2). We compare the features learned by feature adaptation, standard selftraining, and CST. Intuitively, feature adaptation fails because the ideal target solution y = x2[1] x 2 [2] has larger distance in the feature space than other spurious solutions y = x2[1] x 2 [i]. Standard selftraining also fails since it will choose randomly among all solutions. In comparison, CST can recover the ground truth, because it can distinguish the spurious solution resulting in bad pseudo-labels. A classifier trained with those pseudo-labels cannot work on the source domain in turn. This intuition is rigorously justified in the following two theorems. Theorem 3. For ✏ 2 (0, 0.5), the following statements hold for feature adaptation and self-training: • For failure rate ⇠ > 0, and target dataset size nt > ⇥(log 1⇠ ), with probability at least 1 ⇠ over
  the sampling of target data, the solution (✓̂FA, ̂FA) found by feature adaptation satisfies
  ErrQ(✓̂FA, ̂FA) ✏. (14)
  • With probability at least 1 1d 1 , the solution (✓̂ST, ̂ST) of standard self-training satisfies
  ErrQ(✓̂ST, ̂ST) ✏. (15)
  Theorem 4. For failure rate ⇠ > 0, and target dataset size nt > ⇥(log 1⇠ ), with probability at least
sections.heading: 1 ⇠, the solution of CST (̂CST, ✓̂CST) recovers the ground truth of the target dataset:
sections.text: ErrQ(✓̂CST, ̂CST) = 0. (16)
sections.heading: 5 Experiments
sections.text: We test the performance of the proposed method on both vision and language datasets. Cycle SelfTraining (CST) consistently outperforms state-of-the-art feature adaptation and self-training methods. Code is available at https://github.com/Liuhong99/CST.
sections.heading: 5.1 Setup
sections.text: Datasets. We experiment on visual object recognition and linguistic sentiment classification tasks: Office-Home [64] has 65 classes from four kinds of environment with large domain gap: Artistic (Ar), Clip Art (Cl), Product (Pr), and Real-World (Rw); VisDA-2017 [45] is a large-scale UDA dataset with two domains named Synthetic and Real. The datasets consist of over 200k images from 12 categories of objects; Amazon Review [10] is a linguistic sentiment classification dataset of product reviews in four products: Books (B), DVDs (D), Electronics (E), and Kitchen (K). Implementation. We use ResNet-50 [26] (pretrained on ImageNet [53]) as feature extractors for vision tasks, and BERT [16] for linguistic tasks. On VisDA-2017, we also provide results of ResNet101 to include more baselines. We use cross-entropy loss for classification on the source domain. When training the target head ✓̂t and updating the feature extractor with CST, we use squared loss to get the analytical solution of ✓̂t directly and avoid calculating second order derivatives as metalearning [18]. Details on adapting squared loss to multi-class classification are deferred to Appendix B. We adopt SGD with initial learning rate ⌘0 = 2e 3 for image classification and ⌘0 = 5e 4 for sentiment classification. Following standard protocol in [26], we decay the learning rate by 0.1 each 50 epochs until 150 epochs. We run all the tasks 3 times and report mean and deviation in top-1 accuracy. For VisDA-2017, we report the mean class accuracy. Following Theorem 2, we also enhance CST with sharpness-aware regularization [19] (CST+SAM), which help regularize the Lipschitzness of the function class. Due to space limit, we report mean accuracies in Tables 2 and 3 and defer standard deviation to Appendix C.
sections.heading: 5.2 Baselines
sections.text: We compare with two lines of works in domain adaptation: feature adaptation and self-training. We also compare with more complex state-of-the-arts and create stronger baselines by combining feature adaptation and self-training.
  Feature Adaptation: DANN [22], MCD [54], CDAN [37] (which improves DANN with pseudolabel conditioning), MDD [73] (which improves previous domain adaptation with margin theory), Implicit Alignment (IA) [28] (which improves MDD to deal with label shift).
  Self-Training. We include VAT [40], MixMatch [8] and FixMatch [57] in the semi-supervised learning literature as self-training methods. We also compare with self-training methods for UDA: CBST [77], which considers class imbalance in standard self-training, and KLD [78], which improves CBST with label regularization. However, these methods involve tricks specified for convolutional networks. Thus, in sentiment classification tasks where we use BERT backbones, we compare with other consistency regularization baselines: VAT [40], VAT+Entropy Minimization.
  Feature Adaptation + Self-Training. DIRT-T [56] combines DANN, VAT, and entropy minimization. We also create more powerful baselines: CDAN+VAT+Entropy and MDD+Fixmatch.
  Other SOTA. AFN [69] boosts transferability by large norm. STAR [38] aligns domains with stochastic classifiers. SENTRY [48] selects confident examples with a committee of random augmentations.
sections.heading: 5.3 Results
sections.text: Results on 12 pairs of Office-Home tasks are shown in Table 2. When domain shift is large, standard self-training methods such as VAT and FixMatch suffer from the decay in pseudo-label quality. CST outperforms feature adaptation and self-training methods significantly in 9 out of 12 tasks. Note that CST does not involve manually setting confidence threshold or reweighting.
  Table 4 shows the results on VisDA-2017. CST surpasses state-of-the-arts with ResNet-50 and ResNet101 backbones. We also combine feature adaptation and self-training (DIRT-T, CDAN+VAT+entropy and MDD+FixMatch) to test if feature adaptation alleviates the negative effect of domain shift in standard self-training. Results indicate that CST is a better solution than simple combination.
  While most traditional self-training methods include techniques specified for ConvNets such as Mixup [72], CST is a universal method and can directly work on sentiment classification by simply replacing the head and training objective of BERT [16]. In Table 3, most feature adaptation baselines improve over source only marginally, but CST outperforms all baselines on most tasks significantly.
  5.4 Analysis
  Ablation Study. We study the role of each part of CST in self-training. CST w/o Tsallis removes the Tsallis entropy LTsallis,↵. CST+Entropy replaces the Tsallis entropy with standard entropy. FixMatch+Tsallis adds LTsallis,↵ to standard self-training. Observations are shown in Table 5. CST+Entropy performs 3.7% worse than CST, indicating that Tsallis entropy is a better regularization for pseudolabels than standard entropy. CST performs 5.4% better
  than FixMatch, indicating that CST is better adapted to domain shift than standard self-training. While FixMatch+Tsallis outperforms FixMatch, it is still 3.6% behind CST, with much larger total variation distance dTV between pseudo-labels and ground-truths, indicating that CST makes pseudo-labels more reliable than standard self-training under domain shift.
  Quality of Pseudo-labels. We visualize the error of pseudo-labels during training on VisDA-2017 in Figure 5 (Left). The error of target classifier ✓t on the source domain decreases quickly in training, when both the error of pseudo-labels (error of ✓s on Q) and the total variation (TV) distance between pseudo-labels and ground-truths continue to decay, indicating that CST gradually refines pseudolabels. This forms a clear contrast to standard self-training as visualized in Figure 2 (Middle), where the distance dTV remains nearly unchanged throughout training.
  Comparison of Gibbs entropy and Tsallis entropy. We compare the pseudo-labels learned with standard Gibbs entropy and Tsallis entropy on Ar!Cl with ResNet-50 at epoch 40. We compute the difference between the largest and the second largest softmax scores of each target example and plot the histogram in Figure 5 (Right). Gibbs entropy makes the largest softmax output close to 1, indicating over-confidence. In this case, if the prediction is wrong, it can be hard to correct it using self-training. In contrast, Tsallis entropy allows the largest and the second largest scores to be similar.
sections.heading: 6 Related Work
sections.text: Self-Training. Self-training is a mainstream technique for semi-supervised learning [13]. In this work, we focus on pseudo-labeling [52, 31, 2], which uses unlabeled data by training on pseudo-labels generated by a source model. Other lines of work study consistency regularization [4, 51, 55, 40]. Recent works demonstrate the power of such methods [67, 57, 23]. Equipped with proper training techniques, these methods can achieve comparable results as standard training that uses much more labeled examples [17]. Zoph et al. [76] compare self-training to pre-training and joint training. Vu et al. [65], Mukherjee & Awadallah [42] show that task-level self-training works well in few-shot learning. These methods are tailored to semi-supervised learning or general representation learning and do not take domain shift into consideration explicitly. Wei et al. [66], Frei et al. [20] provide the first nice theoretical analysis of self-training based on the expansion assumption.
  Domain Adaptation. Inspired by the generalization error bound of Ben-David et al. [7], Long et al. [34], Zellinger et al. [71] minimize distance measures between source and target distributions to learn domain-invariant features. Ganin et al. [22] (DANN) proposed to approximate the domain distance by adversarial learning. Follow-up works proposed various improvement upon DANN [63, 54, 37, 73, 28]. Popular as they are, failure cases exist in situation like label shift [74, 32], shift in support of domains [29], and large discrepancy between source and target [33]. Another line of works try to address domain adaptation with self-training. Shu et al. [56] improves DANN with VAT and entropy minimization. French et al. [21], Zou et al. [78], Li et al. [32] incorporated various semi-supervised learning techniques to boost domain adaptation performance. Kumar et al. [30], Chen et al. [15] and Cai et al. [11] showed self-training provably works in domain adaptation under certain assumptions.
sections.heading: 7 Conclusion
sections.text: We propose cycle self-training in place of standard self-training to explicitly address the distribution shift in domain adaptation. We show that our method provably works under the expansion assumption and demonstrate hard cases for feature adaptation and standard self-training. Self-training (or pseudolabeling) is only one line of works in the semi-supervised learning literature. Future work can delve into the behaviors of other semi-supervised learning techniques including consistency regularization and data augmentation under distribution shift, and exploit them extensively for domain adaptation.
sections.heading: Acknowledgements
sections.text: This work was supported by the National Natural Science Foundation of China under Grants 62022050 and 62021002, Beijing Nova Program under Grant Z201100006820041, China’s Ministry of Industry and Information Technology, the MOE Innovation Plan and the BNRist Innovation Fund.
title: Cycle Self-Training for Domain Adaptation