## Gradient Harmonization in Unsupervised Domain Adaptation

Fuxiang Huang, Suqi Song, Lei Zhang

Abstract -Unsupervised domain adaptation (UDA) intends to transfer knowledge from a labeled source domain to an unlabeled target domain. Many current methods focus on learning feature representations that are both discriminative for classification and invariant across domains by simultaneously optimizing domain alignment and classification tasks. However, these methods often overlook a crucial challenge: the inherent conflict between these two tasks during gradient-based optimization. In this paper, we delve into this issue and introduce two effective solutions known as Gradient Harmonization, including GH and GH++, to mitigate the conflict between domain alignment and classification tasks. GH operates by altering the gradient angle between different tasks from an obtuse angle to an acute angle, thus resolving the conflict and trade-offing the two tasks in a coordinated manner. Yet, this would cause both tasks to deviate from their original optimization directions. We thus further propose an improved version, GH++, which adjusts the gradient angle between tasks from an obtuse angle to a vertical angle. This not only eliminates the conflict but also minimizes deviation from the original gradient directions. Finally, for optimization convenience and efficiency, we evolve the gradient harmonization strategies into a dynamically weighted loss function using an integral operator on the harmonized gradient. Notably, GH/GH++ are orthogonal to UDA and can be seamlessly integrated into most existing UDA models. Theoretical insights and experimental analyses demonstrate that the proposed approaches not only enhance popular UDA baselines but also improve recent state-of-the-art models.

Index Terms -Unsupervised Domain Adaptation, Gradient Harmonization, Transfer Learning, Image Classification.

✦

## 1 INTRODUCTION

D EEP convolutional neural networks and transformers, driven by extensive labeled samples, have achieved remarkable success in various computer vision tasks such as classification, semantic segmentation and object detection. However, these models often demonstrate high vulnerability when deployed in novel application scenarios due to data distribution discrepancies. The process of collecting and annotating data across various domains is expensive, labor-intensive and time-consuming. Consequently, Unsupervised Domain Adaptation (UDA) arises to transfer the knowledge from a labeled source domain to an unlabeled target domain [15], [37], [56], [65].

In recent years, UDA algorithms have made significant progress in enhancing classification performance [9], [43], [55], [59], [87], [94], [95]. The primary focus of these approaches is to acquire domain-invariant feature representations, thereby achieving domain alignment and narrowing the probability distributions across domains. Currently, domain alignment methods fall into two main categories: distance metrics-based methods [1], [28], [54], [57], [73], [76] and adversarial learning-based methods [4], [8], [14], [20], [44]. The distance metrics-based approaches align the source

- · This work was partially supported by National Key R&amp;D Program of China (2021YFB3100800), National Natural Science Fund of China (62271090), Chongqing Natural Science Fund (cstc2021jcyj-jqX0023). This work is also supported by Huawei computational power of Chongqing Artificial Intelligence Innovation Center. (Corresponding author: Lei Zhang)
- · Fuxiang Huang, Suqi Song and Lei Zhang are with the School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. (E-mail: huangfuxiang@cqu.edu.cn, songsuqi@stu.cqu.edu.cn, leizhang@cqu.edu.cn).

Manuscript received April 19, 2005; revised August 26, 2015.

Fig. 1. Motivation of the proposed Gradient Harmonization. At point a , the black and red arrows point to the optimal gradient descent direction of domain alignment and classification, respectively. The obtuse angle formed by the two gradients of both tasks leads to optimization conflict and further destroy the multi-task optimality.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-85b12716-3056-411a-8ec1-653041052e5e.jpg)

and target domains by mapping them into a shared feature space while minimizing the distribution disparities between them. Inspired by generative adversarial networks [30], adversarial learning techniques have been introduced to tackle domain adaptation challenges [25], [55]. For instance, [25] is among the pioneering methods to achieve domain alignment by introducing a game between the feature extractor and domain classifier. [55] adjusts the adversarial domain adaptation models on discriminative information conveyed in the classifier predictions. These methods usually jointly optimize domain alignment and classification tasks in the course of training.

However, due to their objective differences between the two tasks, the optimal gradient descent directions from the two tasks may be uncoordinated or imbalanced. Therefore, directly sharing network parameters (i.e., feature generator) may result in optimization conflict between tasks and affect domain-invariant feature learning. During optimization, the angle between the gradients of the two tasks is sometimes an obtuse angle, indicating an obvious gradient conflict. As shown in Fig.

1, the black arrow and red arrow point to the optimal gradient descent direction of the domain alignment task and classification task, respectively. In the joint optimization process, obtuse angle formed by the two gradients of both tasks leads to optimization conflict, which reveals the suboptimality of the two tasks.

To further verify the above observations, we conduct some experiments on real cross-domain classification scenarios. Fig. 2 displays inner product distributions of two gradients between two tasks (alignment vs. classification) throughout the training process on task of MNIST → USPS. From Fig. 2, we can obtain the following two observations. First, it implies that the inner product between two gradients is approximately normally distributed with a mean slightly around above zero. Second, both acute and obtuse angles exist on the two models, where obtuse angles account for about 42% of the total number in MCD [70] and 37% in DWL [85] (DWL relieves the imbalance by trade-offing the transferability and discriminability to some extent but ignores optimization conflicts, whereas MCD does not take into account the conflict problem). In the joint optimization process, the aggregated gradient direction generated by two obtuse gradients will be seriously deviated from their respective optimal gradient descent directions, thereby resulting in sub-optimal alignment and classification. Further, we empirically observe that the obtuse angle is always present during training over time and cannot be eliminated automatically. Therefore, how to reasonably eliminate optimization conflicts while maintaining the task-specific optimality during training is a challenging and practical work .

Multi-task learning (MTL), as explored in studies such as [18], [19], [60], [71], [78], focuses on simultaneously optimizing multiple conflicting criteria. These approaches typically seek one or more Pareto optimal solutions that offer different trade-offs. Inspired by MTL, ParetoDA [46] introduces a target-classification-mimicking (TCM) loss based on heldout target data and dynamically seeks a desirable Pareto optimal solution for the target domain. However, Pareto optimization relies on additional information to make decisions and requires significant computational resources and time in high-dimensional or complex systems.

In this paper, to address the aforementioned challenges, we propose an idea of de-conflict on the gradients . Technically, we introduce two intuitive yet highly effective approaches called Gradient Harmonization, involving GH and GH++, which aim to alleviate optimization conflicts that emerge between the domain alignment and classification tasks by leveraging insights from geometric awareness in model optimization. Specifically, we first calculate the original gradients of two tasks to find out if there is a conflict, i.e., the angle between the gradients of the tasks is obtuse (negatively correlated). Subsequently, we employ GH to turn the angle between the original gradients of two tasks from an obtuse angle to an acute angle, i.e., positively correlated. This harmonization of gradients enables both tasks to progress in a coordinated manner. However, this process may cause both tasks to deviate from their original directions to compromise performance. To further mitigate gradient deviation while maintaining the task-specific optimality, we introduce an improved version, GH++, which turns the angle between the original gradients of two tasks from an obtuse angle

Fig. 2. Inner product distributions (histogram) of the two baselines MCD [70] and DWL [85] in the training process. The horizontal axis represents the inner product of the two gradients, and the vertical axis represents frequency (i.e., number of occurrences of inner product of two gradients). Obviously, both (a) and (b) exist obtuse angles, i.e. optimization conflict, and the optimization conflicts of (a) are more serious.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-28a3c7e8-e76a-422f-b678-0dae2cf883e9.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-f59d27f1-a630-4477-9eac-da893fee98cd.jpg)

to a vertical angle, i.e., orthogonal. Then both tasks can evolve harmoniously during joint training while preserving their individual task-specific optimality. Detailed theoretical support is also provided for the proposed approaches.

Furthermore, in order to facilitate optimization and improve computation efficiency, we derive an equivalent but more efficient model of GH/GH++ for unsupervised domain adaptation. The equivalent model is derived as a dynamic objective function, namely UDA+GH/GH++, which can achieve fast elimination of optimization conflicts without sacrificing the task-specific objective. The proposed GH/GH++ is a universal plug-and-play approach for between-task balance learning and can be easily embedded in most alignment-based unsupervised domain adaptation methods. This work is a new upgrade and more general form of our previous CVPR paper [85] from the perspective of gradient harmonization. The main contributions and novelties of this paper are summarized as follows:

- · We verify and visualize the gradient conflict in UDA methods and develop two novel Gradient Harmonization approaches, including GH and GH++. The proposed approaches alleviate the gradient conflict problem between the domain alignment task and classification task by adjusting their gradient angle from obtuse to acute/vertical angle adaptively. Fig. 3 is a representative and illustrative example to depict the UDA model based on GH/GH++.
- · The proposed approaches are orthogonal to most existing UDA methods. To facilitate optimization and improve computation efficiency, we further derive the equivalent models, i.e., UDA with GH/GH++ . Specifically, the proposed approaches can be evolved into dynamically weighted loss functions via an integral operation, and promote optimization.
- · Extensive experiments validate the effectiveness and universality of our approaches on various benchmarks in several mainstream UDA models. More insights and analyses justify the reasonability of the proposed approaches.

## 2 RELATED WORK

## 2.1 Unsupervised Domain Adaptation (UDA)

Unsupervised Domain Adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to

Fig. 3. The usage illustration of our GH module. Optimization objectives include universal domain alignment loss and classification loss. GH module is responsible for harmonization process for the gradients of the two losses. Then the coefficients τ 1 and τ 2 are deduced from GH with the loss gradients g 1 and g 2 to reweight the two losses. Finally, the reweighted loss functions are backpropagated to update network parameters.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-31020471-b1d4-44dc-8cf3-4aef7b4ce1a6.jpg)

solve similar tasks in a new unlabeled domain. Recent works [40], [40], [42], [45], [50], [61], [63], [87], [88], [89] have focused on UDA based on domain alignment and discriminative feature learning methods.

Domain Alignment. The mainstream approach for UDA aims to realize domain alignment by learning domaininvariant feature representations across domains, which can be mainly summarized into two categories: distance metric based methods and adversarial learning based methods. First, the methods based on distance metrics mostly use several existing distance measurement indicators to assess the alignment degree across domains, such as MMD [3], MDD [41] and their variants. With the popularity of deep learning methods, more and more researchers apply deep neural networks to domain adaptation. Compared with traditional non-deep domain adaptation methods, deep methods directly improve the learning effect on different classification tasks. Second, some domain adaptation methods based on adversarial training can learn the domaininvariant feature representations well and better implement knowledge transfer. For example, CDAN [55] proposes an adversarial adaptation model for the discriminative information transmitted in the prediction of the classifier. GVB [14] learns the domain-invariant feature representations by applying the gradually vanishing bridge mechanism on the feature generator. Besides, considering the equilibrium problem of adversarial learning, FGDA [27] reduces the distribution discrepancy by constraining feature gradient. CGDM [21] explicitly minimizes the discrepancy of gradients generated by source samples and target samples to improve the accuracy of target samples. FixBi [63] introduces a fixed ratio-based mixup to augment multiple intermediate domains between the source and target domain.

CAN [38] optimizes the metric for minimizing the domain discrepancy, which explicitly models the intra-class domain discrepancy and the inter-class domain discrepancy. RSDA [31] introduces a spherical classifier for label prediction and a spherical domain discriminator for discriminating domain labels and utilizes robust pseudo-label loss in the spherical feature space. DMRL [84] focuses on guiding the classifier to enhance consistent predictions between samples and enriches the intrinsic structures of the latent space. To combine the source and target domains, DMRL introduces two mixup regularizations based on randomness. Recently, SSRT [74] and TVT [90] learn both transferable and discriminative features by applying Vision Transformer (ViT).

Classification. Other approaches focus on improving the performance of the classification task by enhancing class discriminability feature learning. ETD [43] puts forward an attention-aware optimal transport distance to measure the domain discrepancy under the guidance of the prediction feedback and enables the model to learn distinguished feature representations. MCD [70] proposes to maximize the prediction discrepancy of two classifiers to obtain strong discriminant feature representations. BSP [10] penalizes the largest singular values so that other eigenvectors can be relatively strengthened to boost the feature discriminability.

Balancing Domain Alignment and Classification. Previous approaches concentrate on learning domain-invariant and class-discriminative feature representations by jointly optimizing domain alignment and classification task but ignore imbalance or uncoordinated optimization problem between tasks. Recently, some methods have started to pay attention to this problem and adopt strategies to alleviate it. To adapt to different cross-domain classification scenarios, DWL [85] proposes dynamic weighted learning to avoid the discriminability vanishing problem caused by excessive alignment and domain misalignment problem caused by excessive discriminant learning from a macro perspective, which ignores gradient conflict between two tasks during optimization. Meta [83] maximizes inner product of the gradients of the two tasks during training. However, it cannot guarantee the adjusted gradient direction is close to the original optimal gradient descent direction, and even appears serious deviation. Since Meta processes the gradients of the two tasks in an extreme way, it may produce some negative feedback, such as sacrificing partial alignment or classification performance. Recently, in order to mitigate the effects of conflicting gradients, ParetoDA [46] adopts Pareto optimization [60] to cooperatively optimize all training objectives, which searches for the desirable Pareto optimal solution on the entire Pareto front. However, it is computationally inefficient. In this paper, we propose two intuitive yet efficient approaches, including GH and GH++, to handle the gradient conflict.

Differences. The proposed approaches represent a fundamental departure from previous methods like DWL [85], Meta [83], and ParetoDA [46]. Our primary goal is to implic-

itly address the optimization conflict throughout the entire training process by active gradient harmonization, thereby ensuring a balanced learning between tasks. However, as depicted in Figure 2, the optimization conflict still persists in DWL. Additionally, ParetoDA introduces Pareto optimization and an additional target-classification-mimicking (TCM) loss, which necessitates the involvement of all classwise discriminators. In contrast, the proposed GH/GH++ is evolved into a dynamically weighted loss function via an integral operator on the harmonized gradient.

## 2.2 Multi Task Learning

Multi-task learning (MTL) [6], [19], [62], [78] aims at learning multiple tasks in a unified model to achieve mutual improvement among tasks considering their shared knowledge. By sharing parameters across tasks, MTL methods learn more efficiently with an overall smaller model size compared to learning with separate models. Prior MTL approaches formulate the total objective as a weighted sum of task-specific objectives, such as DWA [53] and GradNorm [11], which optimize weights based on task-specific learning rates or by random weighting. However, these weighted optimization based MLT may not achieve satisfactory performance due to the presence of gradient conflicts among different tasks. Gradient optimization based MLT [18], [48], [51], [60], [71], [93] overcome this limitation, mitigating effects of conflicting or dominating gradients. MGDA [18] proposes to simply update the shared network parameters along a descent direction which leads to solutions that dominate the previous one. PCGrad [93] proposes a 'gradient surgery' to avoid gradient conflicts. CAGrad [48] seeks the gradient direction within the neighborhood of the average gradient and maximizes the worst local improvement of any objective. [2] merge the gradients of auxiliary tasks and applied a scaling factor to adaptively adjust their impact on the main tasks, followed by applying gradient sharing between the main tasks and the merged auxiliary task. GradDrop [12] proposes a probabilistic masking procedure, which samples gradients at an activation layer based on their level of consistency. Aligned-MTL [72] eliminates instability in the training process by aligning the orthogonal components of the linear system of gradients.

In this study, we introduce two simple yet highly efficient approaches, GH and GH++, to effectively address gradient conflicts between any two distinct tasks. Detailed theoretical derivations and fundamental insights regarding the proposed approaches are elaborated in Sections 3. Additionally, under thorough theoretical analysis and a wealth of experiments, we have substantiated the soundness and effectiveness of the proposed methodologies.

## 3 PROPOSED APPROACH

## 3.1 Problem Definition

Given a labeled source domain D s = { x , y s i s i } n s i =1 with n s samples and an unlabeled target domain D t = { x t j } n t j =1 with n t samples, where y s i is the class label of the i th source sample x s i . D s and D t share the same feature space and category space, but have different data distributions. Our purpose is to utilize the labeled data D s and unlabeled data D t to learn a deep model, which can accurately predict the class label of samples in the target domain.

## 3.2 A General Framework of UDA

Adversarial learning has proven to be an effective method for domain alignment, starting from Domain Adversarial Neural Network (DANN) [25]. The basic idea is to trick the domain discriminator D by generating features via a feature generator G . Then the domain discriminator predicts whether the generated feature by G is from the source domain or the target domain. The training of domain alignment is achieved through the game between generator and domain discriminator. The parameter θ g of generator G and the parameter θ d of domain discriminator D are optimized by the following domain alignment objective function.

$$\mathcal { L } _ { d o m } ( \theta _ { g }, \theta _ { d } ) = \mathbb { E } _ { x _ { i } ^ { s } \sim \mathcal { D } _ { s } } l o g [ D ( G ( x _ { i } ^ { s } ) ) ] + \\ \mathbb { E } _ { x _ { j } ^ { t } \sim \mathcal { D } _ { t } } l o g [ 1 - D ( G ( x _ { j } ^ { t } ) ) ].$$

In order to improve the classification performance of the target domain samples, we must first ensure that the classifier C can correctly classify the samples from the source domain. Thus, the supervised classification loss can be described as

$$\Gamma _ { i } \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phile \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phis \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \PhI \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \PhiPhone \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Philon \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phicon \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phj \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phs \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phim \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phip \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Ph } \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Psi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Psim \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \pmi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Pr \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \Psi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \Psi\quad \Phi \quad \Phi \quad \Phi \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \quad \Phi \$$

where L ce is the standard cross-entropy loss function.

During training stage, the existing methods usually jointly optimize the two objective functions ( L dom and L cls ) to obtain domain-invariant and class-discriminant feature representation. The overall minimax objective function is

$$\min _ { \theta _ { g }, \theta _ { c } } \, \max _ { \theta _ { d } } \, \mathcal { L } _ { d o m } + \mathcal { L } _ { c l s },$$

where θ g , θ d , θ c denote the parameters of feature generator, domain discriminator and classifier, respectively.

## 3.3 Gradient Harmonization (GH)

Domain alignment and classification are two different tasks. Their optimal gradient descent directions may not be coordinated, which results in the optimization conflict of the two loss functions in the training process and deteriorates the final domain adaptation performance.

In order to ensure that the two target tasks can be optimized in a coordinated manner, we propose an idea of de-conflict on the gradients, which then formulates the proposed GH technique, i.e., altering the gradient angle between different tasks from an obtuse angle to an acute angle. The de-conflict process for the gradients (i.e., g 1 and g 2 ) is schematically shown in Fig. 4. Specifically, we first provide three Lemmas to support our idea. Then the proposed GH is summarized as Theorem 1 . For convenience, we define L 1 (Θ) and L 2 (Θ) as the general form of any two conflicted loss functions. In UDA, L 1 (Θ) and L 2 (Θ) represent the domain alignment loss L dom ( θ , θ g d ) and the classification loss L cls ( θ , θ g c ) , respectively. Θ = { θ , θ g d , θ c } indicates model parameters of generator, discriminator and classifier. In fact, the three Lemmas aim to deduce the gradient harmonization formula during model optimization (training) phase, by exhausted mathematical solving process.

Lemma 1. Given two objective functions L 1 (Θ) and L 2 (Θ) , we define g 1 and g 2 as their gradient, respectively, and ˜ g 1 is the result of harmonizing the gradient g 1 . For minimizing the

Fig. 4. Overall idea of de-conflict for the gradients g 1 and g 2 of two tasks. (a) displays the angle between two gradients is an acute angle. (b) displays the angle between two gradients is an obtuse angle. Following GH, (b) needs to be processed and (b1), (b2) and (b3) are harmonization process. (b1) and (b2) are the details of performing our gradient harmonization on g 1 and g 2 , resp. (b3) is final harmonization results. ˜ g 1 and ˜ g 2 represent the gradients after harmonization, resp. Finally, after applying GH, the angle between g 1 and g 2 has changed from obtuse angle to acute angle.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-7cba8d74-3f09-43ce-98f6-c23833cc3c32.jpg)

objective L 1 (Θ -˜ )+ g 1 L 2 (Θ -˜ ) g 1 , we consider to first minimize the objective L 1 (Θ -˜ ) g 1 , and then there is,

∗ ∗

$$\tilde { g } _ { 1 } = g _ { 1 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } } g _ { 2 }, \quad ( 4 ) \quad \text{Eq. (1} } \quad \text{sum}$$

$$\ n i n i m i z e \quad \text{duality holds, i.e., $d^{*}=p^{*}$, we have} \\ \alpha _ { 1 } \tilde { g } _ { 1 } ^ { T } g _ { 1 } = 0, \alpha _ { 2 } \tilde { g } _ { 1 } ^ { T } g _ { 2 } = 0.$$

where δ ( ) · represents the indicator function whose value is 0 or 1 and the mathematical expression is:

$$\delta ( A ) = \begin{cases} 1, \ i f \ A \ i s$$

Proof. In order to minimize the loss function L 1 (Θ -˜ ) g 1 , the problem is transformed into solving the following model:

$$\min _ { \tilde { g } _ { 1 } } \ \mathcal { L }$$

Due to the gradient g 1 = ∇ L Θ 1 (Θ) is the fastest descent direction of the objective function L 1 (Θ) , if the harmonic gradient ˜ g 1 is the same as g 1 as much as possible, the loss function L 1 (Θ) will be optimized along the gradient descent direction. If the angle between the harmonic gradient ˜ g 1 and g 1 is an acute angle, then ˜ g 1 points to the direction of gradient descent. Naturally, the angle between ˜ g 1 and g 2 = ∇ L Θ 2 (Θ) is also expected to be an acute angle. Therefore, the above model is transformed into:

$$\sum \nolimits \sum \nolimits \sum \nolimits \$$

The problem (7) is a convex optimization problem. To solve the problem (7), we define the Lagrangian function

$$\begin{array} {$$

where α 1 , α 2 are the Lagrangian multipliers associated with the inequality constraint of problem (7). To simplify the solution, first note that

$$P & = \max _ { \alpha _ { 1 } \geq 0, \alpha _ {$$

This means that we can express the optimal value of the primal problem (7) as:

$$p ^ { * } = \min _ { \tilde { g } _ { 1 } } \ \max _ { \alpha _ { 1 }$$

By the definition of the dual function, we can express the optimal value of the dual problem:

$$d ^ { * } = \max _ { \alpha _ { 1 } \geq 0, \alpha _ { 2 } \geq 0 }$$

where α 1 ≥ 0 , α 2 ≥ 0 is the dual feasibility condition. It is easy to get the weak duality d ∗ ≤ p ∗ . When the strong

Eq. (12) is known as complementary slackness condition. To summarize, for convex optimization problem (7), we have

$$\begin{cases} \begin{array} { c } \nabla _ { \tilde { g } _ { 1 } } \math$$

Conditions (13) are called the Karush-Kuhn-Tucker (KKT) conditions [5], which is the the necessary condition for pair of primal and dual optimal points. We can solve the dual problem Eq. (11). As for the inner optimization problem, i.e., D = min ˜ g 1 L (˜ g , α 1 1 , α 2 ) , we then obtain its optimal solution by setting its derivative with respect to ˜ g 1 as zero. Then,

$$\tilde { g } _ { 1 } = g _ { 1 } + \alpha _ { 1 } g _ { 1 } + \alpha _$$

We can calculate the extremum of D by substituting Eq. (14) into Eq. (8) :

$$\colon g _ { 1 } \quad \begin{array} {$$

We then get the outer optimization problem by substituting Eq. (15) into Eq. (11) as follows:

$$\max _ { \alpha _ { 1 } \geq 0, \alpha _ { 2 } \geq 0 } - \frac { 1 } { 2 } \| g _ { 1 } + \alpha _ { 1 } g _ { 1 } + \alpha _ { 2 } g _ { 2 } \| ^ { 2 }. \quad \ ( 1 6 )$$

For the problem (16), we can take the derivative of α 1 and α 2 respectively, and set the derivative to 0. Then we can obtain the optimal solution: α ∗ 1 = -1 , α ∗ 2 = 0 . Obviously, the obtained α ∗ 1 does not satisfy the dual feasibility condition. Therefore, it is necessary to find the boundary solution that satisfies the constraints, which is established under two possible cases:

Case 1: Suppose α ∗ 1 = 0 , the Eq. (16) is transformed into:

$$\begin{smallmatrix} \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \Phi & \$$

We can set the derivative of α 2 to 0, i.e., ( g 1 + α g 2 2 ) T g 2 = 0 . Then we can obtain the optimal solution:

$$\alpha _ { 2 } ^ { * } = - \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } }.$$

Obviously, Eq. (18) satisfies the KKT conditions 1 ⃝⃝⃝ 2 4 in Eq. (13). According to dual feasibility, i.e., 3 ⃝ in Eq. (13), α ∗ 2 = -g T 1 g 2 ∥ g 2 ∥ 2 ≥ 0 , then we have g T 1 g 2 ≤ 0 . Therefore, we can obtain a boundary solution

$$\begin{cases} \alpha _ { 1 } ^ { * } = 0, & \quad \varsigma _ { 1 } ^ { * } \\ \alpha _ { 2 } ^ { * } = - \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } }, & g _ { 1 } ^ { T } g _ { 2 } \leq 0 \\ \quad \quad \quad$$

Case 2: Suppose α ∗ 2 = 0 , the Eq. (16) is transformed into:

$$\min _ { \alpha _ { 1 } } \frac { 1 } { 2 } \| g _ { 1 } + \alpha _ { 1 } g _ { 1 } \| ^ { 2 }. \quad \quad \ ( 2 0 ) \, \stackrel { ( b ) \, a } { \text{gate} }$$

Then we can obtain α ∗ 1 = 0 in nature. Obviously, it satisfies the KKT conditions 1 ⃝⃝⃝ 3 4 in Eq. (13). According to primal feasibility, i.e., 2 ⃝ in Eq. (13), we have g T 1 g 2 ≥ 0 . Therefore, the boundary solution is

$$\begin{cases} \alpha _ { 1 } ^ { * } = 0, & \\ \alpha _ { 2 } ^ { * } = 0, & g _ { 1 } ^ { T } g _ { 2 } \geq 0 & \end{cases} \quad \begin{cases} ( 2 1 ) & \quad i \end{cases}$$

In summary, the optimal solution of problem (16) is:

$$\begin{cases} \mu _ { 1 } ^ { * } = 0 & \mu _ { 2 } ^ { * } = - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } }. \\ \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \ddots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \\ \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \\ \\ \end{cases}$$

Then the optimal value of the primal problem (7) ˜ g can be obtained by substituting Eq. (22) into Eq. (14).

$$\tilde { g } _ { 1 } = g _ { 1 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { \tilde { g } _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } } g _ { 2 }, \quad \ ( 2 3 ) \quad \tilde { g } = \tilde { g } _ { 1 } \\ \tilde { \ }. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \.. \.. \.. \.. \.. \.. \........................................................................................................................................................................................................ \.................................................................................................................................................................................................... \.. \................................................................................................................................................................................................. \..... \................................................................................................................................................................................................. ....................................................................................................................................................................................................... ..... ................................................................................................................................................................................................. ..... \................................................................................................................................................................................................. \.. .................................................................................................................................................................................................... .. .................................................................................................................................................................................................... \.. .. ................................................................................................................................................................................................. .. .. \.................................................................................................................................................................................................... .. \................................................................................................................................................................................................. \.. \.. \.............................................................................................................................................................................................. \........ \.............................................................................................................................................................................................. \..... \.. \.............................................................................................................................................................................................. \.. \..... \.............................................................................................................................................................................................. \.. \.. \.. \........................................................................................................................................................................................... \........... \.......................................................................................................................................................$$

where ˜ g 1 is the result of harmonizing the gradient g 1 of L 1 . Then the proof of Eq. (4) in Lemma 1 is completed.

The above process not only solves the problem (7) but also harmonizes the gradient g 1 of L 1 (Θ) , i.e., Fig. 4 (b1).

Lemma 2. Define ˜ g 2 as the result of harmonizing the gradient g 2 , we consider to further minimize the optimization objective L 2 (Θ -˜ ) g 2 , then there is,

$$\tilde { g } _ { 2 } = g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 2 } ^ { T } g _ { 1 } } { \| g _ { 1 } \| ^ { 2 } } g _ { 1 }. \quad \quad ( 2 4 ) \quad \text{Theo}$$

Proof. In order to minimize the loss function L 2 (Θ -˜ ) g 2 , similar to Lemma 1 , we need to solve the following minimization problem.

$$\min _ { \tilde { g } } \mathcal { L } _ { 2 } ( \Theta - \tilde { g } _ { 2 } ) \\ s. t. \, \mathcal { L } _ { 1 } ( \Theta - \tilde { g } _ { 2 } ) \leq \mathcal { L } _ { 1 } ( \Theta ), \mathcal { L } _ { 2 } ( \Theta - \tilde { g } _ { 2 } ) \leq \mathcal { L } _ { 2 } ( \Theta ). \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
- - \quad s. t. \, \mathcal { L } _ { 1 } ( \Theta - \tilde { g } _ { 2 } ) \leq \mathcal { L } _ { 2 } ( \Theta ). \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
- - \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

The above model can be transformed into

$$\underset { \tilde { g } _ { 2 } } { \text{move can we main} } \, \underset { \tilde { g } _ { 2 } } { \text{in} } \, \frac { 1 } { 2 } \| g _ { 2 } - \tilde { g } _ { 2 } \| ^ { 2 } & & ( 2 6 ) \, \underset { \tilde { g } _ { 2 } ^ { T } } { \text{algo} } \, \\ s. t. \, \tilde { g } _ { 2 } ^ { T } g _ { 1 } \geq 0, \tilde { g } _ { 2 } ^ { T } g _ { 2 } \geq 0. & & \text{this} \, \mathbb { 1 } \\ \text{between} \, \underset { \tilde { g } _ { 2 } ^ { T } \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots \, \int \dots.$$

Similar to problem (7), by solving (26), we can obtain

$$\tilde { \ g } _ { 2 } = g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { \tilde { g } _ { 2 } ^ { T } g _ { 1 } } { \| g _ { 1 } \| ^ { 2 } } g _ { 1 }, \quad \ \ ( 2 7 ) \quad \text{inter} \\ \tilde { \ g } _ { 3 } \. \. \. \. \. \. \. \. \. \. \. \ \ \ \. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

where ˜ g 2 is the result of harmonizing the gradient g 2 of L 2 (Θ) , i.e., Fig. 4 (b2).

Lemma 3. Define the overall loss function L = L 1 (Θ)+ L 2 (Θ) of two tasks, and ˜ g denotes the overall harmonized gradient. We

Fig. 5. Gradient aggregation. (a) Gradient aggregation when the angle of the original gradients g 1 and g 2 is an acute angle. (b) Gradient aggregation when the angle of the original gradients g 1 and g 2 is an obtuse angle. (c) Gradient aggregation after GH for case (b). Comparing (b) and (c), GH changes the magnitude and the direction of the aggregated/combined gradient g and gradient harmonization is realized.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-1b280acc-d803-4509-8215-5de3e3f03b3a.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-162fe941-530a-4e44-bdb5-aebce96f129e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-04de441e-d7bb-48a8-a9b5-7d7914c39a24.jpg)

Fig. 6. The Essence of Gradient Harmonization. (a) and (b) denote the essence illustration of performing GH on g 1 and g 2 , respectively.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-49853634-e32a-4323-bbc6-d7073daee49b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-fbf3d1f3-d700-4da7-b072-6b54745e24cc.jpg)

consider the whole optimization objective L 1 (Θ -˜)+ g L 2 (Θ -˜) g , then there is,

˜=˜ + ˜

g

g

1

g

2

$$\int _ { \int \mathcal { L } _ { 1 } } ^ { ( \leftarrow \int \mu ) } \mu ^ { \mu } \mu ^ { \mu } \mu ^ { \mu } = & g _ { 1 } + g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } } g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 2 } ^ { T } g _ { 1 } } { \| g _ { 1 } \| ^ { 2 } } g _ { 1 }. \\ & \quad \, \int _ { \int \mathcal { L } _ { 1 } } ^ { ( \leftarrow \int \mu ) } \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \alpha \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \sin \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \cos \lim \beta \tan \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \osigma \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \exists \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \sec \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \decoration \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \less�$$

Proof. According to Lemma 1 and Lemma 2 , the results of harmonizing the gradients g 1 and g 2 , i.e., ˜ g 1 and ˜ g 2 , can be obtained. By aggregating Eq. (4) in Lemma 1 and Eq. (24) in Lemma 2, the whole gradient after harmonization can be obtained as Eq. (28), and proof of Lemma 3 is completed.

For brevity, we summarize the above process of GH in Theorem 1 .

Theorem 1. For any two different tasks, optimization conflicts can be eliminated by Gradient Harmonization (GH). Define the overall loss function L (Θ) = L 1 (Θ) + L 2 (Θ) , composed of two sub-objectives (refer to domain alignment and classification in this paper), and g 1 and g 2 as the gradient of L 1 (Θ) and L 2 (Θ) . Then the overall harmonized gradient ˜ g of the whole loss L (Θ) with GH can be formulated as:

$$\tilde { g } = g _ { 1 } + g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } } g _ { 2 } - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 2 } ^ { T } g _ { 1 } } { \| g _ { 1 } \| ^ { 2 } } g _ { 1 }. \ ( 2 9 )$$

Obviously, in the optimization process of standard SGD algorithm, the overall gradient of the loss L is g = g 1 + g 2 . In this paper, two cases are considered based on the correlation between gradients g 1 and g 2 .

Case I: When g 1 and g 2 are positively correlated or unrelated, i.e., cos( g , g 1 2 ) ≥ 0 or g T 1 g 2 ≥ 0 , the angle between the two gradients is an acute or vertical angle. Then, we believe that there is no conflict between the two objectives in training, that is, two gradients are coordinated or balanced. Thus g 1 and g 2 do not need to be harmonized.

Case II: When g 1 and g 2 are negatively correlated, i.e., cos( g , g 1 2 ) &lt; 0 or g T 1 g 2 &lt; 0 , the angle between the two gradients is an obtuse angle. Then, we believe that there is a gradient optimization conflict between the two objectives.

In this case, g 1 and g 2 need to be harmonized. For the overall loss function L (Θ) = L 1 (Θ) + L 2 (Θ) of an arbitrary model, the proposed UDA+GH is to find a gradient ˜ g that satisfies the following problem:

$$\min _ { \tilde { g } } \ \mathcal { L } _ { 1 } ( \Theta - \tilde { g } ) + \mathcal { L } _ { 2 } ( \Theta - \tilde { g } ), \quad \quad ( 3 0 ) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

where Eq. (30) can be solved by aforementioned three lemmas. Firstly, the harmonic gradient ˜ g 1 of g 1 can be obtained by Lemma 1 . Then the harmonic gradient ˜ g 2 of g 2 can be obtained by Lemma 2 . Finally, as can be seen from Lemma 3 , it is easy to acquire the whole aggregated gradient g in Eq. (29) after harmonization and the gradient aggregation process is visualized in Fig. 5.

## 3.4 Essence and Insights of GH

To understand the essence of GH more intuitively, we demonstrate the proposed gradient harmonization by backward inference in this section. Fig. 6 (a) and (b) show the essential analysis by performing gradient harmonization on original gradients g 1 and g 2 , respectively. Note that we mainly consider the case where gradient conflict exists, that is, the angle between g 1 and g 2 is an obtuse angle.

First, we analyze the essence of gradient harmonization for g 1 (i.e., Fig. 6 (a)). θ denotes the angle between original gradients g 1 and g 2 . n is the projection of g 1 on the reversely extended line of g 2 . m is perpendicular to g 2 . Note that each letter except θ on the figure represents a vector. Naturally, m can be derived as Eq. (31), from which we can see that the expression of the vector m is the same as Eq. (4) when g T 1 g 2 &lt; 0 . That is, the vector m is the result of gradient harmonization of g 1 (i.e., ˜ g 1 ).

$$\breve { \omega } \sim \tilde { \ker } \text{(n-n)} \, \stackrel { \cdot } { (i.e., \tilde { g } _ { 1 } ) }. \quad & \quad \text{denote} \\ m = g _ { 1 } - n = & g _ { 1 } - | g _ { 1 } | \cdot \cos ( \pi - \theta ) \cdot ( - \frac { g _ { 2 } } { | g _ { 2 } | } ) & \quad \text{the su} \\ = & g _ { 1 } - | g _ { 1 } | \cdot \cos \theta \cdot \frac { g _ { 2 } } { | g _ { 2 } | } & \quad \text{th} \quad \text{the tw} \\ = & g _ { 1 } - | g _ { 1 } | \cdot \frac { \langle g _ { 1 }, g _ { 2 } \rangle } { | g _ { 1 } | \cdot | g _ { 2 } | } \cdot \frac { g _ { 2 } } { | g _ { 2 } | } & \quad \text{the derivat} \\ = & g _ { 1 } - \frac { \langle g _ { 1 }, g _ { 2 } \rangle } { \| g _ { 2 } \| ^ { 2 } } g _ { 2 } \cdot & \quad \text{min} \\ \text{where} \, \langle \cdot \rangle \, \text{is the inner product operator.} } \text{$(with$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$\quad$$$

where ⟨·⟩ is the inner product operator.

Second, we analyze the essence of gradient harmonization for g 2 (i.e., Fig. 6 (b)). Similar to (a), the vector e is the projection of g 2 on the reversely extended line of g 1 . The vector f is perpendicular to g 1 . Through the derivation of Eq. (32), it can be found that the expression of the vector f is the same as Eq. (24) when g T 1 g 2 &lt; 0 . That is, the vector f is the result of harmonizing the gradient g 2 (i.e., ˜ g 2 ).

$$\text{is the result of harmonizing the gradient } g _ { 2 } ( i. e., \tilde { g } _ { 2 } ). \quad \text{the or!} \\ f = g _ { 2 } - e = & g _ { 2 } - | g _ { 2 } | \cdot \cos ( \pi - \theta ) \cdot ( - \frac { g _ { 1 } } { | g _ { 1 } | } ) \quad \text{and } \widehat { B } \\ = & g _ { 2 } - | g _ { 2 } | \cdot \cos \theta \cdot \frac { g _ { 1 } } { | g _ { 1 } | } \quad \text{and } \widehat { g } _ { 1 } \text{ and } \tilde { \tilde { g } } _ { 1 } \text{ as } ] \text{$\tilde { g } _ { 1 } \text{, with} \\ = & g _ { 2 } - | g _ { 2 } | \cdot \frac { \langle g _ { 1 }, g _ { 2 } \rangle } { | g _ { 1 } | \cdot | g _ { 2 } | } \cdot \frac { g _ { 1 } } { | g _ { 1 } | } \quad \text{and } \text{$\text{$\text{$single}$ to Fig.} \\ = & g _ { 2 } - \frac { \langle g _ { 1 }, g _ { 2 } \rangle } { \| g _ { 1 } \| ^ { 2 } } g _ { 1 }. \quad \text{and } \tilde { g } _ { 2 } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{\quad } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text { } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text{ } \text$$

From the above derivations, the nature of GH is summarized as follows. 1) The harmonic gradient ˜ g 1 is essentially the projection of the raw gradient g 1 on the vertical line of g 2 . 2) The harmonic gradient ˜ g 2 is essentially the projection of the raw gradient g 2 on the vertical line of g 1 .

Therefore, with the above nature, the following three observations can be obtained. 1) The proposed gradient harmonization method not only ensures that the harmonic gradient ( ˜ g /g 1 ˜ 2 ) is close to the original gradient ( g 1 / g 2 ), but also ensures that the aggregated gradients before and after applying GH are close to each other. 2) If the angle between the original gradients g 1 and g 2 is θ , then the angle between harmonized gradient ˜ g 1 and ˜ g 2 becomes π -θ after applying GH. That is, GH really realizes the transformation from obtuse angle to acute angle. 3) The proposed GH aims to move the harmonic gradients towards the direction favorable for optimization, rather than those arbitrary gradient directions with acute angles.

## 3.5 Improved Version: GH++

In this section, we propose an improved version called GH++, which aims to adjust the gradient angle between the two tasks from an obtuse angle to a vertical angle, i.e., making them orthogonal. This is to eliminate the conflict but simultaneously relieve the gradient deviation. Fig. 7 visually illustrates the concepts of the proposed GH and GH++, with a primary focus on scenarios where gradient conflict exists, i.e., when the angle between g 1 and g 2 is obtuse.

Gradient deviation. For clarity, we designate the original gradients as g 1 = -→ OA and g 2 = - -→ OB . Let θ represent the angle between g 1 and g 2 . As illustrated in Fig. 7 (a), we denote the gradients after applying GH as ˜ g 1 = - -→ OC and ˜ g 2 = - -→ OD , where OC ⊥ OB and OD ⊥ OA . Apparently, the sum of the angles deviated from the original directions is 2( θ -π 2 ) , i.e., ∠ AOC + ∠ DOB . In other words, although GH can promote the positive correlation between the two gradients, its optimization gradient is seriously deviated from the original gradient. Therefore, we propose an improved version, GH++, to eliminate the conflict and minimize the sum of the gradient deviations. As shown in Fig. 7 (b), GH++ adjusts the gradient angle between the two tasks from an obtuse angle to a vertical angle, i.e., ∠ EOF . The sum of the gradient deviation angles is ( θ -π 2 ) i.e., ∠ AOE + ∠ FOB , which is half of the sum of the gradient deviations of GH.

Specifically, as shown in Fig. 7 (b), let denote the harmonized gradients of GH++ as ˜ g 1 = - -→ OE and ˜ g 2 = - -→ OF . In order to resolve the conflict and relieve the deviation from the original gradient directions, we designate OE ⊥ OF , i.e., ∠ EOF = π 2 , where points E and F move along arcs ˜ AC and ¯ BD , respectively. Intuitively, △ EOF forms a rotatable right triangle. We define the direction of rotation from g 1 to ˜ g 1 as positive. β represents the rotating angles from g 1 to ˜ g 1 , which is a positive angle and ¯ β represents the rotating angles from g 2 to ˜ g 2 , which is a negative angle. According to Fig. 7 (b), the harmonization gradients of GH++, i.e., ˜ g 1 and ˜ g 2 , can be represented as

$$\tilde { g } _ { 1 } & = \overrightarrow { O E } = \overrightarrow { O A } + \overrightarrow { A E } = g _ { 1 } + \overrightarrow { A E }, \\ \tilde { g } _ { 2 } & = \overrightarrow { O F } = \overrightarrow { O B } + \overrightarrow { B F } = g _ { 2 } + \overrightarrow { B F }.$$

Fig. 7. Illustration of the proposed GH/GH++. The blue and green arrows represent the directions of original gradients and harmonization gradients, respectively. Let θ denote the angle between original gradients, i.e., g 1 and g 2 . (a) GH turns the angle between the original gradients from an obtuse angle to an acute angle. The sum of the gradient deviation angles is 2( θ -π 2 ) , i.e., ∠ AOC + ∠ DOB . (b) GH++ turns the angle between the original gradients from an obtuse angle to a vertical angle. The sum of the gradient deviation angles is ( θ -π 2 ) i.e., ∠ AOE + ∠ FOB .

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-a3a13108-bd86-45bd-b076-b64ee53fee0b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-814de10b-d1af-422f-ab63-42f84c3186e3.jpg)

Since the △ AOE and △ BOF are isosceles triangles, we can obtain -→ AE and - -→ BF according to the trigonometric formula, whose expression can be written as

$$\overrightarrow { A E } & = 2 \cdot \overrightarrow { O A } \cdot \sin \frac { \not \, A O E } { 2 } = 2 \cdot g _ { 1 } \cdot \sin \frac { \beta } { 2 }, \\ \overrightarrow { B F } & = 2 \cdot \overrightarrow { O B } \cdot \sin \frac { \not \, B O F } { 2 } = 2 \cdot g _ { 2 } \cdot \sin \frac { \bar { \beta } } { 2 }. \\ \text{Substitution} \, \text{F} \, \left ( 3 4 \right ) \text{into} \, \text{F} \, \left ( 3 3 \right ) \text{we can derive the harm-} \quad \dots 1 \, \vdots.$$

Substituting Eq. (34) into Eq. (33), we can derive the harmonized gradients of GH++.

$$\tilde { g } _ { 1 } & = ( 1 + 2 \cdot \sin \frac { \beta } { 2 } ) \cdot g _ { 1 }, & \quad \text{erat} \\ \tilde { g } _ { 2 } & = ( 1 + 2 \cdot \sin \frac { \bar { \beta } } { 2 } ) \cdot g _ { 2 }. & \quad \text{ally} \, \cdot \\ \text{to Fixture 7} \, \text{$\theta$} \, \text{we have a variable anole $\beta$} \, = \, \text{proa}$$

Referring to Figure 7 (b), we have a variable angle β = ∠ AOE ∈ [0 , θ -π 2 ] , where θ = ∠ AOB = arccos g T 1 g 2 | g 1 |·| g 2 | ∈ ( π 2 , π ] . To simplify, we can express β as follows:

$$\beta = \lambda \triangle A O C = \lambda ( \theta - \frac { \pi } { 2 } ) = \lambda ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ), \ ( 3 6 ) \quad \text{the orig}$$

where λ ∈ [0 , 1] serves as a trade-off parameter, controlling the magnitude of deviation from ˜ g 1 to g 1 . When λ = 0 , ˜ g 1 = g 1 = -→ OA remains unchanged, and g 2 = - -→ OD . When λ = 1 , g 1 = - -→ OC , and ˜ g 1 = g 2 = - -→ OB remains constant. Since OE ⊥ OF , i.e., ∠ EOF = π 2 , ¯ β can be represented as

$$\bar { \beta } = \zeta B O F = - \zeta F O B = - ( \theta - \frac { \pi } { 2 } - \beta ) = \beta - ( \theta - \frac { \pi } { 2 } ). \ ( 3 7 ) \quad \text{if } \mathbb { G H } + \cdot \cdot \cdot$$

Substituting Eq. (36) into Eq. (37), we have

$$\text{substituting} \, & \text{$\mathcal{ }Eq.\, (\mathfrak { s } )$ into $\mathcal{ }Eq.\, (\mathfrak { s } )$, we have} \\ & \bar { \beta } = \lambda ( \theta - \frac { \pi } { 2 } ) + ( \theta - \frac { \pi } { 2 } ) = ( \lambda - 1 ) ( \theta - \frac { \pi } { 2 } ) \\ & = ( \lambda - 1 ) ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ). \\ & \text{$\mathcal{ }T_{1}=1$} \\ \text{Finalize without the $\mathcal{ }Fq.\, (\mathfrak { s } )$ and $\mathcal{ }Eq.\, (\mathfrak { s } )$} \, \text{$\mathcal{ }u$}$$

Finally, substituting Eq. (36) and Eq. (38) into Eq. (35), we can obtain the harmonization gradients of GH++ as follows:

$$\begin{array} { l } \text{can obtain the narmonization gradients or $\mathcal{P1++ as follows:}$} & \text{bedded} \\ \tilde { g } _ { 1 } = ( 1 + 2 \cdot \sin \frac { \lambda ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ) } { 2 } ) \cdot g _ { 1 }, & \text{on the} \\ \tilde { g } _ { 2 } = ( 1 + 2 \cdot \sin \frac { ( \lambda - 1 ) ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ) } { 2 } ) \cdot g _ { 2 }. & \tilde { \mathcal { L } } \vdots \\ \text{Similar to GH. when there is no conflict. i.e. } a _ { 2 } ^ { T } a _ { 2 } > 0. & \text{where} \end{array}$$

Similar to GH, when there is no conflict, i.e., g T 1 g 2 ≥ 0 , no action. The proposed GH++ is indicated as Theorem 2 .

Theorem 2. For any two different tasks, optimization conflicts can be eliminated by GH++. Define the overall loss function L (Θ) = L 1 (Θ) + L 2 (Θ) , composed of two sub-objectives (refer to domain alignment and classification in this paper). Define g 1 and g 2 as the gradient of L 1 (Θ) and L 2 (Θ) , respectively, then the general expression of the overall harmonized gradient ˜ g of the whole loss L (Θ) with GH++ can be formulated as:

$$\tilde { g } = \tilde { g } _ { 1 } + \tilde { g } _ { 2 }$$

$$& g = & g _ { 1 } + g _ { 2 } \\ \text{arrows} \\ \text{grad} \\ \text{ts, i.e.,} \\ \text{th,i.e.,} \\ \text{form an} \\ \text{angles} \\ \text{between} \\ \text{even} \\ \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text	$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text
$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text                                                                                                                                                                                                         }                                                                                                                                                                                                       
                                                                                                                                                                                                        $                                                                                                                                                                                                        -                                                                                                                                                                                                        +                                                                                                                                                                                                       .                                                                                                                                                                                                       ,                                                                                                                                                                                                        "                                                                                                                                                                                                        <                                                                                                                                                                                                        >                                                                                                                                                                                                        </                                                                                                                                                                                                        :                                                                                                                                                                                                        ;$$

It is worth noting that GH++ presents a more favorable optimization scenario for the original task compared to GH. This advantage arises from the fact that the sum of the gradient deviation from the original direction of GH++ is only half of GH.

## 3.6 Equivalent Model of UDA with GH/GH++

By combining the general UDA model (Eq. (3), described in Section 3.2) with gradient harmonization strategies (Eq. (29)/Eq. (40), a well-balanced UDA model can be trained and implemented. However, the gradient aggregation operator in Eq. (29)/Eq. (40) is intricate and has an impact on optimization efficiency. Therefore, we introduce a computation-efficient alternative model that is functionally equivalent to UDA with GH/GH++. For convenience, we can express the proposed gradient harmonization approaches, i.e., Eq. (29)/Eq. (40), as follows:

$$\tilde { g } = \tau _ { 1 } g _ { 1 } + \tau _ { 2 } g _ { 2 },$$

where τ 1 and τ 2 are constants that can be calculated by using the original gradients g 1 and g 2 . Notably, the gradient g 1 of the original loss L 1 (Θ) and the gradient g 2 of the original loss L 2 (Θ) can be easily computed, only if the UDA model is fixed. Then, if GH is chosen to resolve the conflict, τ 1 and τ 2 can be calculated as follows.

$$\text{e calculate as rolows.} \\ \tau _ { 1 } & = 1 - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 2 } ^ { T } g _ { 1 } } { \| g _ { 1 } \| ^ { 2 } }, \\ \tau _ { 2 } & = 1 - \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \frac { g _ { 1 } ^ { T } g _ { 2 } } { \| g _ { 2 } \| ^ { 2 } }. \\ \text{$+$ is used to eliminate the conflict $\tau$ and $\tau$ can be}$$

If GH++ is used to eliminate the conflict, τ 1 and τ 2 can be calculated as follows.

$$( 3 8 ) & \text{ calculated as follows.} \\ \tau _ { 1 } = ( 1 + 2 \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \sin \frac { \lambda ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ) } { 2 } ), \\ \tau _ { 1 } = ( 1 + 2 \delta ( g _ { 1 } ^ { T } g _ { 2 } < 0 ) \sin \frac { ( \lambda - 1 ) ( \arccos \frac { g _ { 1 } ^ { T } g _ { 2 } } { | g _ { 1 } | \cdot | g _ { 2 } | } - \frac { \pi } { 2 } ) } { 2 } ). \\ 5 ), \, \text{we} \quad & \text{Ultimately, we can derive the equivalent UDA model em-}$$

Ultimately, we can derive the equivalent UDA model embedded GH/GH++ by conducting the integral operation on the gradient in Eq. (41). The overall loss of UDA with GH/GH++ model can be represented as

$$\tilde { \mathcal { L } } = \int ( \tau _ { 1 } g _ { 1 } + \tau _ { 2 } g _ { 2 } ) d \Theta = \tau _ { 1 } \mathcal { L } _ { 1 } ( \Theta ) + \tau _ { 2 } \mathcal { L } _ { 2 } ( \Theta ), \quad ( 4 4 )$$

where L 1 (Θ) and L 2 (Θ) can be the domain alignment loss and classification loss described in Eq. (1) and Eq. (2),

## Algorithm 1: Balanced UDA with GH/GH++

input

: Source samples { x , y } s , Target samples output: Optimal parameters

s i s i n i =1 { x t j } n t j =1 , Optimal parameters Θ = { θ , θ g d , θ c } , learning rate η , max iteration, λ (if GH++) Θ

- 1 Initialization Optimal parameters Θ ; repeat
- 2 Compute the domain alignment loss L dom ( θ , θ g d ) , i.e., L 1 (Θ) , and the classification loss L cls ( θ , θ g c ) , i.e., L 2 (Θ) ;
- 3 Compute original gradients g 1 and g 2 ;
- 4 Calculate the inner product of two gradients, i.e., g T 1 g 2 , and the indicator function by Eq. (5);
- 5 Compute τ 1 and τ 2 by Eq. (42)/Eq. (43);
- 6 Compute updated total loss

L

˜ =

τ

1

L

dom

(

θ , θ

g

d

) +

τ

2

L

cls

- 7 Update model parameters:

8

Θ

t

+1

←-

Θ

t

-

η

▽

Θ

t

(

θ , θ

g

˜

L

- 9 until max iteration is reached ;

respectively, for unsupervised domain adaptation. Finally, the objective function of the UDA with GH/GH++ is:

$$\min _ { \theta _ { g }, \theta _ { c } } \, \max _ { \theta _ { d } } \, \tilde { \mathcal { L } } = \tau _ { 1 } \mathcal { L } _ { d o m } ( \theta _ { g }, \theta _ { d } ) + \tau _ { 2 } \mathcal { L } _ { c l s } ( \theta _ { g }, \theta _ { c } ). \quad ( 4 5 ) \, \stackrel { \overbrace { \overbrace { \overbrace { \overbrace { G V B } [ 1 4 } } } } { \overbrace { \overbrace { \overbrace { \overbrace { G V B } [ 1 4 } } } }$$

It can be seen that the proposed approaches essentially reweight the original loss function L 1 (Θ) and L 2 (Θ) . The computation of the weights τ 1 and τ 2 needs to use the gradient of the original loss function, as presented in Eq. (42)/Eq. (43). Finally, a balanced and efficient UDA model is formulated. Without loss of generality, when g T 1 g 2 &gt; 0 , we can observe τ 1 =1 and τ 2 =1, and the overall loss function is degenerated into ˜ = L L 1 (Θ)+ L 2 (Θ) , i.e., the general UDA model. We describe the training procedure in Algorithm 1.

Notably, the proposed approaches are plug-and-play modules and can be deployed in almost all models with different losses L 1 (Θ) and L 2 (Θ) . In order to clearly observe the computational cost of the equivalent model, we discuss the training speed and recognition accuracy before and after applying GH in Table 6.

## 4 EXPERIMENTS

## 4.1 Datasets

Office-31 [69] is a mainstream benchmark dataset for visual domain adaptation, which consists of three distinct domains: Amazon (A), DSLR (D), Webcam (W). It totally contains 4,652 images from 31 categories. We evaluate our method in all 6 different transfer tasks across domains.

Office-Home [79] is a more challenging and harder benchmark than Office-31. It contains 15.5K images across 65 object categories from 4 different domains: Artistic images (Ar), Clip Art (Cl), Product images (Pr), and Real-World images (Rw). We evaluate our method in all 12 different transfer tasks across domains.

Digits Datasets . We mainly study three datasets: MNIST (M) [91], USPS (U) [36] and SVHN (S) [64]. MNIST and USPS are two general handwriting recognition datasets involving 10 categories. SVHN is obtained from house numbers in

c

)

;

TABLE 1

Accuracy ( % ) on Office-31 for UDA (ResNet50). Avg ‡ represents the mean values except W ↔ D. Note that TVT and SSRT exploit the ViT backbone pre-trained on ImageNet-21K, while TVT ∗ and SSRT ∗ indicate that their ViT backbone is pre-trained on ImageNet-1K.

| Method         | A → W   | A → D   | W → A   | W → D   | D → A   | D → W   | Avg   | Avg ‡   |
|----------------|---------|---------|---------|---------|---------|---------|-------|---------|
| ResNet-50 [32] | 68.4    | 68.9    | 60.7    | 99.3    | 62.5    | 96.7    | 76.1  | 65.1    |
| JAN [58]       | 85.4    | 84.7    | 70.0    | 99.8    | 68.6    | 97.4    | 84.3  | 77.2    |
| CAT [17]       | 91.1    | 90.6    | 66.5    | 99.6    | 70.4    | 98.6    | 86.1  | 79.7    |
| ETD [43]       | 92.1    | 88.0    | 67.8    | 100.0   | 71.0    | 100.0   | 86.2  | 79.7    |
| TAT [49]       | 92.5    | 93.2    | 72.1    | 100.0   | 73.1    | 99.3    | 88.4  | 82.7    |
| TADA [82]      | 94.3    | 91.6    | 73.0    | 99.8    | 72.9    | 98.7    | 88.4  | 83.0    |
| SymNets [96]   | 90.8    | 93.9    | 72.5    | 100.0   | 74.6    | 98.8    | 88.4  | 83.0    |
| BNM [13]       | 92.8    | 92.9    | 73.8    | 100.0   | 73.5    | 98.8    | 88.6  | 83.3    |
| ALDA [8]       | 95.6    | 94.0    | 72.5    | 100.0   | 72.2    | 97.7    | 88.7  | 83.6    |
| MDD [97]       | 94.5    | 93.5    | 72.2    | 100.0   | 74.6    | 98.4    | 88.9  | 83.7    |
| SAFN [88]      | 88.8    | 87.7    | 67.9    | 99.8    | 69.8    | 98.4    | 85.4  | 78.6    |
| Meta [83]      | 93.9    | 91.6    | 74.1    | 100.0   | 73.7    | 98.7    | 88.7  | 83.3    |
| SHOT [47]      | 90.1    | 94.0    | 74.3    | 100.0   | 74.7    | 98.4    | 88.6  | 83.3    |
| CAN [38]       | 94.5    | 95.0    | 77.0    | 100.0   | 78.0    | 99.1    | 90.6  | 86.1    |
| FixBi [63]     | 96.1    | 95.0    | 79.4    | 100.0   | 78.7    | 99.3    | 91.4  | 87.3    |
| FGDA [27]      | 93.3    | 93.2    | 72.7    | 100.0   | 73.2    | 99.1    | 88.6  | 83.1    |
| ParetoDA [46]  | 95.0    | 95.4    | 75.7    | 100.0   | 77.6    | 98.9    | 90.4  | 85.9    |
| TVT ∗ [90]     | 95.7    | 95.4    | 80.3    | 100.0   | 80.6    | 98.7    | 91.8  | 88.0    |
| TVT [90]       | 96.4    | 96.4    | 86.1    | 100.0   | 84.9    | 99.4    | 93.9  | 90.9    |
| CDAN [55]      | 94.1    | 92.9    | 69.3    | 100.0   | 71.0    | 98.6    | 87.7  | 81.8    |
| CDAN+ GH       | 94.7    | 94.0    | 72.0    | 100.0   | 72.6    | 98.6    | 88.6  | 83.3    |
| CDAN+ GH++     | 95.5    | 94.4    | 73.3    | 100.0   | 73.0    | 98.8    | 89.2  | 84.0    |
| MCD [70]       | 88.6    | 92.2    | 69.7    | 100.0   | 69.5    | 98.5    | 86.5  | 80.0    |
| MCD+ GH        | 91.4    | 92.2    | 70.6    | 100.0   | 69.9    | 98.6    | 87.1  | 81.0    |
| MCD+ GH++      | 90.8    | 92.8    | 71.2    | 100.0   | 70.7    | 98.6    | 87.4  | 81.4    |
| DWL [85]       | 89.2    | 91.2    | 69.8    | 100.0   | 73.1    | 99.2    | 87.1  | 80.8    |
| DWL+ GH        | 89.2    | 91.1    | 69.9    | 100.0   | 73.7    | 99.3    | 87.2  | 81.0    |
| DWL+ GH++      | 90.4    | 91.4    | 70.5    | 100.0   | 73.7    | 99.3    | 87.5  | 81.5    |
| GVB [14]       | 94.8    | 95.0    | 73.7    | 100.0   | 73.4    | 98.7    | 89.3  | 84.2    |
| GVB+ GH        | 94.9    | 95.4    | 74.0    | 100.0   | 75.3    | 99.0    | 89.8  | 84.9    |
| GVB+ GH++      | 95.1    | 95.2    | 74.1    | 100.0   | 75.8    | 99.1    | 89.9  | 85.0    |
| SSRT ∗ [74]    | 97.7    | 98.6    | 82.2    | 100.0   | 83.5    | 99.2    | 93.5  | 90.5    |
| SSRT ∗ + GH    | 98.5    | 98.6    | 83.6    | 100.0   | 84.2    | 99.3    | 94.0  | 91.2    |
| SSRT ∗ + GH++  | 98.9    | 98.8    | 83.3    | 100.0   | 84.6    | 99.3    | 94.1  | 91.4    |
| SSRT [74]      | 98.4    | 99.2    | 84.0    | 100.0   | 85.3    | 99.2    | 94.4  | 91.7    |
| SSRT+ GH       | 98.9    | 99.2    | 84.2    | 100.0   |         | 99.2    | 94.5  | 92.0    |
|                |         | 99.6    |         |         | 85.6    |         |       |         |
| SSRT+ GH++     | 98.7    |         | 84.6    | 100.0   | 85.8    | 99.4    | 94.7  | 92.2    |

Google Street View images. We conduct experiments in 3 universal tasks, including M → U, U → Mand S → M.

VisDA-2017 [68] is a simulation-to-real dataset for domain adaptation with two distinct domains: synthetic object images rendered from 3D models and real object images. It contains over 280K images across 12 categories.

DomainNet [67] is the largest domain adaptation dataset consisting of about 600K images from six distinct domains, including Clipart (clp), Infograph (inf), Painting (pnt), Quickdraw (qdr), Real (rel), Sketch (skt). There are 48K-172K images categorized into 345 classes per domain. We evaluate our method in all 30 cross-domain tasks.

## 4.2 Implementation Details

We compare our method with the following state-of-theart unsupervised domain adaptation methods: ResNet [32], DAN (Deep Adaptation Networks) [54], DANN (Domainadversarial Neural Networks) [25], JAN (Joint Adaptation Networks) [58], DRCN (Deep Reconstruction-Classification Networks) [29], CoGAN (Coupled Generative Adversarial Networks) [52], ADDA (Adversarial Discriminative Domain Adaptation) [75], CyCADA ( Cycle-consistent Adversarial Domain Adaptation) [33], CAT (Cluster Alignment with a Teacher) [17], TPN (Transferrable prototypical networks) [66], LWC (Light-weight Calibrator) [92], ETD (Enhanced Transport Distance) [43], TAT (Transferable Adversarial Training) [49], TADA (Transferable Attention

TABLE 2

Accuracy ( % ) on Office-Home for UDA (ResNet-50). Avg ‡ represents the mean values except Pr ↔ Rw. Note that TVT and SSRT exploit the ViT backbone pre-trained on ImageNet-21K, while TVT ∗ and SSRT ∗ indicate the ViT backbone is pre-trained on ImageNet-1K.

| Method              | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Ar → ClAr → PrAr → RwCl → ArCl → PrCl → RwPr → ArPr → ClPr → RwRw → ArRw → ClRw → Pr   | Avg       | Avg ‡     |
|---------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-----------|-----------|
| ResNet-50 [32]      | 34.9                                                                                   | 50.0                                                                                   | 58.0                                                                                   | 37.4                                                                                   | 41.9                                                                                   | 46.2                                                                                   | 38.5                                                                                   | 31.2                                                                                   | 60.4                                                                                   | 53.9                                                                                   | 41.2                                                                                   | 59.9                                                                                   | 46.1      | 43.3      |
| JAN [58]            | 45.9                                                                                   | 61.2                                                                                   | 68.9                                                                                   | 50.4                                                                                   | 59.7                                                                                   | 61.0                                                                                   | 45.8                                                                                   | 43.4                                                                                   | 70.3                                                                                   | 63.9                                                                                   | 52.4                                                                                   | 76.8                                                                                   | 58.3      | 55.3      |
| DAN [54]            | 43.6                                                                                   | 57.0                                                                                   | 67.9                                                                                   | 45.8                                                                                   | 56.5                                                                                   | 60.4                                                                                   | 44.0                                                                                   | 43.6                                                                                   | 67.7                                                                                   | 63.1                                                                                   | 51.5                                                                                   | 74.3                                                                                   | 56.3      | 53.3      |
| DANN [25]           | 45.6                                                                                   | 59.3                                                                                   | 70.1                                                                                   | 47.0                                                                                   | 58.5                                                                                   | 60.9                                                                                   | 46.1                                                                                   | 43.7                                                                                   | 68.5                                                                                   | 63.2                                                                                   | 51.8                                                                                   | 76.8                                                                                   | 57.6      | 54.6      |
| TAT [49]            | 51.6                                                                                   | 69.5                                                                                   | 75.4                                                                                   | 59.4                                                                                   | 69.5                                                                                   | 68.6                                                                                   | 59.5                                                                                   | 50.5                                                                                   | 76.8                                                                                   | 70.9                                                                                   | 56.6                                                                                   | 81.6                                                                                   | 65.8      | 63.2      |
| TADA [82]           | 53.1                                                                                   | 72.3                                                                                   | 77.2                                                                                   | 59.1                                                                                   | 71.2                                                                                   | 72.1                                                                                   | 59.7                                                                                   | 53.1                                                                                   | 78.4                                                                                   | 72.4                                                                                   | 60.0                                                                                   | 82.9                                                                                   | 67.6      | 65.0      |
| SymNets [96]        | 47.7                                                                                   | 72.9                                                                                   | 78.5                                                                                   | 64.2                                                                                   | 71.3                                                                                   | 74.2                                                                                   | 64.2                                                                                   | 48.8                                                                                   | 79.5                                                                                   | 74.5                                                                                   | 52.6                                                                                   | 82.7                                                                                   | 67.6      | 64.9      |
| ALDA [8]            | 53.7                                                                                   | 70.1                                                                                   | 76.4                                                                                   | 60.2                                                                                   | 72.6                                                                                   | 71.5                                                                                   | 56.8                                                                                   | 51.9                                                                                   | 77.1                                                                                   | 70.2                                                                                   | 56.3                                                                                   | 82.1                                                                                   | 66.6      | 64.0      |
| SAFN [88]           | 58.9                                                                                   | 76.2                                                                                   | 81.4                                                                                   | 70.4                                                                                   | 73.0                                                                                   | 77.8                                                                                   | 72.4                                                                                   | 55.3                                                                                   | 80.4                                                                                   | 75.8                                                                                   | 60.4                                                                                   | 79.9                                                                                   | 71.8      | 70.2      |
| SHOT [47]           | 57.1                                                                                   | 78.1                                                                                   | 81.5                                                                                   | 68.0                                                                                   | 78.2                                                                                   | 78.1                                                                                   | 67.4                                                                                   | 54.9                                                                                   | 82.2                                                                                   | 73.3                                                                                   | 58.8                                                                                   | 84.3                                                                                   | 71.8      | 69.5      |
| FixBi [63]          | 58.1                                                                                   | 77.3                                                                                   | 80.4                                                                                   | 67.7                                                                                   | 79.5                                                                                   | 78.1                                                                                   | 65.8                                                                                   | 57.9                                                                                   | 81.7                                                                                   | 76.4                                                                                   | 62.9                                                                                   | 86.7                                                                                   | 72.7      | 70.4      |
| FGDA [27]           | 52.3                                                                                   | 77.0                                                                                   | 78.2                                                                                   | 64.6                                                                                   | 75.5                                                                                   | 73.7                                                                                   | 64.0                                                                                   | 49.5                                                                                   | 80.7                                                                                   | 70.1                                                                                   | 52.3                                                                                   | 81.6                                                                                   | 68.3      | 65.7      |
| ParetoDA [46]       | 56.8                                                                                   | 75.9                                                                                   | 80.5                                                                                   | 64.4                                                                                   | 73.5                                                                                   | 73.7                                                                                   | 65.6                                                                                   | 55.5                                                                                   | 81.3                                                                                   | 75.2                                                                                   | 61.1                                                                                   | 83.9                                                                                   | 70.6      | 68.2      |
| TVT ∗ [90]          | 67.1                                                                                   | 83.5                                                                                   | 87.3                                                                                   | 77.4                                                                                   | 85.0                                                                                   | 85.6                                                                                   | 75.6                                                                                   | 64.9                                                                                   | 86.6                                                                                   | 79.1                                                                                   | 67.2                                                                                   | 88.0                                                                                   | 78.9      | 77.3      |
| TVT [90]            | 74.9                                                                                   | 86.8                                                                                   | 89.5                                                                                   | 82.8                                                                                   | 88.0                                                                                   | 88.3                                                                                   | 79.8                                                                                   | 71.9                                                                                   | 90.1                                                                                   | 85.5                                                                                   | 74.6                                                                                   | 90.6                                                                                   | 83.6      | 82.2      |
| CDAN [55]           | 50.7                                                                                   | 70.6                                                                                   | 76.0                                                                                   | 57.6                                                                                   | 70.0                                                                                   | 70.0                                                                                   | 57.4                                                                                   | 50.9                                                                                   | 77.3                                                                                   | 70.9                                                                                   | 56.7                                                                                   | 81.6                                                                                   | 65.8      | 63.1      |
| CDAN+ GH            | 52.4                                                                                   | 74.2                                                                                   | 78.7                                                                                   | 62.3                                                                                   | 72.3                                                                                   | 73.4                                                                                   | 61.6                                                                                   | 51.4                                                                                   | 80.7                                                                                   | 73.1                                                                                   | 57.3                                                                                   | 82.3                                                                                   | 68.3      | 65.7      |
| CDAN+ GH++          | 54.3                                                                                   | 73.9                                                                                   | 78.7                                                                                   | 62.4                                                                                   | 73.7                                                                                   | 73.6                                                                                   | 61.8                                                                                   | 53.5                                                                                   | 80.6                                                                                   | 73.3                                                                                   | 57.7                                                                                   | 83.5                                                                                   | 68.9      | 66.3      |
| MCD [70]            | 48.9                                                                                   | 68.3                                                                                   | 74.6                                                                                   | 61.3                                                                                   | 67.6                                                                                   | 68.8                                                                                   | 57.0                                                                                   | 47.1                                                                                   | 75.1                                                                                   | 69.1                                                                                   | 52.2                                                                                   | 79.6                                                                                   | 64.1      | 61.5      |
| MCD+ GH             | 52.1                                                                                   | 71                                                                                     | 77.6                                                                                   | 62.7                                                                                   | 68.7                                                                                   | 70.5                                                                                   | 60.9                                                                                   | 51.9                                                                                   | 78.8                                                                                   | 74.2                                                                                   | 60.0                                                                                   | 82.1                                                                                   | 67.5      | 65.0      |
| MCD+ GH++           | 52.2                                                                                   | 72.3                                                                                   | 77.6                                                                                   | 62.6                                                                                   | 69.8                                                                                   | 70.4                                                                                   | 62.8                                                                                   | 53.1                                                                                   | 78.6                                                                                   | 74.7                                                                                   | 60.3                                                                                   | 81.9                                                                                   | 68.0      | 65.6      |
| DWL [85]            | 46.6                                                                                   | 67.9                                                                                   | 74.6                                                                                   | 57.7                                                                                   | 66.4                                                                                   | 68.4                                                                                   | 58.5                                                                                   | 46.0                                                                                   | 76.1                                                                                   | 69.9                                                                                   | 51.8                                                                                   | 78.7                                                                                   | 63.6      | 60.8      |
| DWL+ GH             | 47.2                                                                                   | 69.5                                                                                   | 75.9                                                                                   | 59.4                                                                                   | 67.3                                                                                   | 68.4                                                                                   | 59.5                                                                                   | 46.9                                                                                   | 76.6                                                                                   | 70.1                                                                                   | 51.9                                                                                   | 79.7                                                                                   | 64.4      | 61.6      |
| DWL+ GH++           | 47.5                                                                                   | 69.5                                                                                   | 76.1                                                                                   | 59.5                                                                                   | 67.2                                                                                   | 69.1                                                                                   | 60.2                                                                                   | 47.0                                                                                   | 76.4                                                                                   | 70.5                                                                                   | 53.0                                                                                   | 79.6                                                                                   | 64.6      | 62.0      |
| GVB [14]            | 57.0                                                                                   | 74.7                                                                                   | 79.8                                                                                   | 64.6                                                                                   | 74.1                                                                                   | 74.6                                                                                   | 65.2                                                                                   | 55.1                                                                                   | 81.0                                                                                   | 74.6                                                                                   | 59.7                                                                                   | 84.3                                                                                   | 70.4      | 67.9      |
| GVB+ GH             | 57.3                                                                                   | 75.4                                                                                   | 79.9                                                                                   | 65.3                                                                                   | 74.5                                                                                   | 75.1                                                                                   | 65.9                                                                                   | 55.4                                                                                   | 81.6                                                                                   | 74.6                                                                                   | 60.1                                                                                   | 84.3                                                                                   | 70.8      | 68.3      |
| GVB+ GH++           | 57.7                                                                                   | 75.5                                                                                   | 80.2                                                                                   | 65.1                                                                                   | 75.2                                                                                   | 75.4                                                                                   | 65.6                                                                                   | 55.4                                                                                   | 81.2                                                                                   | 74.7                                                                                   | 60.4                                                                                   | 84.4                                                                                   | 70.9      | 68.5      |
| SSRT ∗ [74]         | 75.2                                                                                   | 89.0                                                                                   | 91.1                                                                                   | 85.1                                                                                   | 88.3                                                                                   | 90.0                                                                                   | 85.0                                                                                   | 74.2                                                                                   | 91.3                                                                                   | 85.7                                                                                   | 78.6                                                                                   | 91.8                                                                                   | 85.4      | 84.2      |
| SSRT ∗ + GH         | 75.4                                                                                   | 90.0                                                                                   | 91.3                                                                                   | 85.5                                                                                   | 89.1                                                                                   | 90.1                                                                                   | 85.8                                                                                   | 75.1                                                                                   | 91.3                                                                                   | 86.8                                                                                   | 78.9                                                                                   | 92.2                                                                                   | 85.9      | 84.8      |
| SSRT ∗ + GH++       | 75.5                                                                                   | 89.6                                                                                   | 91.4                                                                                   | 86.1                                                                                   | 89.4                                                                                   | 90.3                                                                                   | 85.4                                                                                   | 75.2                                                                                   | 91.3                                                                                   | 86.9                                                                                   | 79.7                                                                                   | 92.4                                                                                   | 86.1      | 85.0      |
| SSRT [74]           | 76.0                                                                                   | 88.7                                                                                   | 91.2                                                                                   | 85.2                                                                                   | 88.7                                                                                   | 90.2                                                                                   | 85.0                                                                                   | 74.6                                                                                   | 91.4                                                                                   | 86.3                                                                                   | 78.4                                                                                   | 92.5                                                                                   | 85.7      | 84.4      |
| SSRT+ GH SSRT+ GH++ | 76.5 76.3                                                                              | 89.5 89.7                                                                              | 91.3 91.6                                                                              | 85.8 86.5                                                                              | 89.2 89.4                                                                              | 90.2 90.4                                                                              | 85.2 85.4                                                                              | 75.3 75.3                                                                              | 91.8 91.3                                                                              | 87.1 87.5                                                                              | 78.7 79.2                                                                              | 92.8 92.7                                                                              | 86.1 86.3 | 84.9 85.1 |

for Domain Adaptation) [82], SymNets (Symmetric Networks) [96], BNM (Batch Nuclear-norm Maximization) [13], ALDA (Adversarial-learned Loss for Domain Adaptation) [8], MIMTFL (Mutual Information Maximisation and Transferable Feature Learning) [26], Meta (Metaalign) [83], MDD (Margin Disparity Discrepancy) [97], CDAN [55], MCD (Maximum Classification Discrepancy) [70], GVB (Gradually Vanishing Bridge) [14] and DWL (Dynamic Weighted Learning) [85], SAFN (Stepwise Adaptive Feature Norm) [88], SWD (Sliced Wasserstein Discrepancy) [40], DMRL (Dual Mixup Regularized Learning) [84], CAN (Contrastive Adaptation Network) [38], FixBi [63], FGDA (Feature Gradient Distribution Alignment) [27] and ParetoDA (Pareto Domain Adaptation) [46], CGDM (Crossdomain Gradient Discrepancy Minimization) [21], SSRT (Safe Self-Refinement for Transformer-based domain adaptation) [74] and TVT (Transferable Vision Transformer) [90]. SSRT and TVT are transformer-based approaches, while other approaches are CNN-based. We use the results in their original papers for fair comparison. When there is no relevant experimental result in the original papers, we implement them according to their released source codes.

Furthermore, we investigate the effect of GH/GH++ on five popular UDA approaches, including four CNNbased approaches (i.e., CDAN [55], MCD [70], GVB [14] and DWL [85]) and one transformer-based approach (i.e., SSRT [74]). For CNN-based approaches, we use pre-trained ResNet-50 [32] on ImageNet-1K [16] as backbone network for Office-31 and Office-Home, and adopt the pre-trained

ResNet-101 [32] as backbone network for VisDA-2017 and DomainNet. We employ LeNet [39] as backbone for Digits datasets. For transformer-based approach, SSRT uses the ViT-base with 16 × 16 patch size pre-trained on ImageNet21K and ImageNet-1K (denoted by SSRT ) as the trans-∗ former backbones, respectively. In terms of optimization, we follow the original protocol of the baselines. Note that, in this paper, CDAN means CDAN+E in [55]) and GVB refers to GVB-GD in [14]. For GH++, we empirically set λ to 0.5. MCD achieves domain alignment by the game between two classifiers and the generator. Therefore, the gradient of the discrepancy loss between the two classifiers is just the gradient of domain alignment loss. We implement our method in PyTorch by following UDA protocol [58] that source domain is labeled and target domain is unlabeled. For fair comparisons, we independently run all experiments five times and report the average target accuracy.

## 4.3 Results on UDA

Tables 1, 2, 3, 4 and 5 present evaluation results on Office-31, Office-Home, VisDA-2017, Digits and DomainNet, respectively. Generally, the transformer-based results, such as TVT and SSRT, are much better than CNN-based results, which has been validated in previous work. For TVT and SSRT, the ViT backbone pre-trained on ImageNet-1K is slightly inferior than ImageNet-21K (i.e., TVT ∗ &amp;TVT, SSRT &amp;SSRT). ∗ Considering that the CNNs are pre-trained on ImageNet-1K, we take SSRT ∗ for fair analysis as default in the following.

TABLE 3 Accuracy ( % ) on Digits Datasets for UDA (LeNet). ∗ indicates that the backbone is ViT pre-trained on ImageNet-1K.

| Method        |   M → U |   U → M | S → M   | Avg   |
|---------------|---------|---------|---------|-------|
| DAN [54]      |    80.3 |    77.8 | 73.5    | 77.2  |
| DRCN [29]     |    91.8 |    73.7 | 82.0    | 82.5  |
| CoGAN [52]    |    91.2 |    89.1 | -       | -     |
| ADDA [75]     |    89.4 |    90.1 | 76.0    | 85.2  |
| CyCADA [33]   |    95.6 |    96.5 | 90.4    | 94.2  |
| CAT [17]      |    90.6 |    80.9 | 98.1    | 89.9  |
| TPN [66]      |    92.1 |    94.1 | 93.0    | 93.1  |
| LWC [92]      |    95.6 |    97.1 | 97.1    | 96.6  |
| ETD [43]      |    96.4 |    96.3 | 97.9    | 96.9  |
| SWD [40]      |    98.1 |    97.1 | 98.9    | 98.0  |
| SHOT [47]     |    91.9 |    96.8 | 89.6    | 92.8  |
| TVT ∗ [90]    |    97.7 |    98.9 | 98.0    | 98.2  |
| TVT [90]      |    98.2 |    99.4 | 99.0    | 98.9  |
| CDAN [55]     |    95.6 |    98   | 89.2    | 94.3  |
| CDAN+ GH      |    96.8 |    98.1 | 90.6    | 95.2  |
| CDAN+ GH++    |    96.5 |    98.3 | 9 0.7   | 95.2  |
| MCD [70]      |    94.2 |    94.1 | 96.2    | 94.8  |
| MCD+ GH       |    96.7 |    96.8 | 97.5    | 97.0  |
| MCD+ GH++     |    97.3 |    97.1 | 97.1    | 97.2  |
| DWL [85]      |    97.3 |    97.4 | 98.1    | 97.6  |
| DWL+ GH       |    98.7 |    98.5 | 98.8    | 98.7  |
| DWL+ GH++     |    98.9 |    98.6 | 99.1    | 98.9  |
| GVB [14]      |    96.3 |    95.1 | 90.0    | 93.8  |
| GVB+ GH       |    96.7 |    95.4 | 91.0    | 94.4  |
| GVB+ GH++     |    96.9 |    95.5 | 91.7    | 94.7  |
| SSRT ∗ [74]   |    98.4 |    99.4 | 99.1    | 99.0  |
| SSRT ∗ + GH   |    99.5 |    99.4 | 99.2    | 99.3  |
| SSRT ∗ + GH++ |    99.4 |    99.5 | 99.2    | 99.3  |
| SSRT [74]     |    98.6 |    99.4 | 99.2    | 99.0  |
| SSRT+ GH      |    98.9 |    99.4 | 99.3    | 99.2  |
| SSRT+ GH++    |    99.2 |    99.5 | 99.3    | 99.3  |

In order to demonstrate the effectiveness and universality of the proposed approaches in UDA, we investigate the effect of GH/GH++ on five popular UDA approaches, including CNN-based approaches (i.e., CDAN, MCD, GVB and DWL) and transformer-based approach (i.e., SSRT). We can observe that the proposed approaches can further improve the baseline, whether based on transformer or CNN. This is attributed to the proposed approaches in alleviating gradient conflict between domain alignment task and classification task in optimization. Meanwhile, it verifies that the proposed approaches can be easily plugged and played in the existing UDA methods. In addition, GH++ usually performs better than GH. The reason is that during removing the gradient conflict between the two tasks, GH++ minimizes the gradient deflection, such that the two tasks can evolve harmoniously during joint training while preserving their individual task-specific optimality.

Specifically, as shown in Table 1, when GH/GH++ is applied, CDAN, MCD, GVB, DWL and SSRT outperform the original models by 1.5%/2.2%, 1.0%/1.4%, 0.2%/0.7%, 0.7%/0.8% and 0.7%/0.9% on Avg ‡ of Office-31, respectively. As shown in Table 2, CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 2.6%/3.2%, 4.5%/5.1%, 0.8%/1.2%, 0.4%/0.6% and 0.6%/0.8% on Avg ‡ of Office-Home, respectively. As shown in Table 3, we can observe that CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 0.9%/0.9%, 2.2%/2.4%, 1.1%/1.3%, 0.6%/0.9% and 0.3%/0.3% on Avg of Digits, respectively. As shown in Table 4, CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original mod-

TABLE 4 Accuracy ( % ) on VisDA-2017 for UDA (ResNet-101). ∗ indicates that the backbone is ViT pre-trained on ImageNet-1K.

| Method          |   Synthetic → Real (Avg) |
|-----------------|--------------------------|
| ResNet-101 [32] |                     52.4 |
| JAN [58]        |                     61.6 |
| DAN [54]        |                     61.1 |
| DANN [25]       |                     57.4 |
| SAFN [88]       |                     76.1 |
| SWD [40]        |                     76.4 |
| DMRL [84]       |                     75.5 |
| SHOT [47]       |                     82.9 |
| CAN [38]        |                     87.2 |
| FixBi [63]      |                     87.2 |
| CGDM [21]       |                     82.3 |
| ParetoDA [46]   |                     83.2 |
| TVT ∗ [90]      |                     85.1 |
| TVT [90]        |                     86.7 |
| CDAN [55]       |                     73.9 |
| CDAN+ GH        |                     75.3 |
| CDAN+ GH++      |                     77.8 |
| MCD [70]        |                     71.9 |
| MCD+ GH         |                     76   |
| MCD+ GH++       |                     76.4 |
| DWL [85]        |                     77.1 |
| DWL+ GH         |                     77.6 |
| DWL+ GH++       |                     77.9 |
| GVB [14]        |                     77   |
| GVB+ GH         |                     77.7 |
| GVB+ GH++       |                     78.3 |
| SSRT ∗ [74]     |                     88.8 |
| SSRT ∗ + GH     |                     89.5 |
| SSRT ∗ + GH++   |                     89.4 |
| SSRT [74]       |                     88.9 |
| SSRT+ GH        |                     89.2 |
| SSRT+ GH++      |                     89.6 |

els by 1.4%/3.9%, 4.1%/4.5%, 0.5%/0.8%, 0.7%/0.3% and 0.7%/0.6% on VisDA-2017, respectively. From the results in Table 5, we can observe that CDAN, MCD, GVB, DWL and SSRT after applying GH/GH++ outperform the original models by 3.2%/3.7%, 7.1%/7.3%, 0.7%/1.0%, 1.0%/1.3% and 1.1%/1.4% on Avg of DomainNet, respectively. The proposed approaches generally improve CDAN and MCD more than other baselines because the original models of CDAN and MCD have more obvious optimization conflicts than other baselines. Fig. 2 also confirms this perspective. Therefore, the proposed approaches have better performance when the inherent gradient conflict is serious. Besides, it is worth noting that the proposed approaches usually achieve greater performance gains on large-scale datasets, i.e., VisDA-2017 and DomainNet, which further indicate the effectiveness and versatility of our method.

## 5 MODEL ANALYSIS AND DISCUSSION

## 5.1 Feature Visualization

Fig. 8 describes the t-SNE [77] visualizations of features learned by MCD (baseline) and MCD+GH on the tasks of U → Mand M → U. Fig. 8 (a) and (c) are visualization features generated by MCD. Fig. 8 (b) and (d) are visualization features generated by MCD+GH. It can be observed that both features learned by MCD and MCD+GH achieve wellperformed global alignment effect with 10 clusters under two tasks. Further, the visualization feature distributions with GH deployed have better clustering effect and have fewer samples distributed across class boundaries, which intuitively boosts the feature discriminability. In addition,

TABLE 5

Accuracy ( % ) on DomainNet for UDA (ResNet-101). In each sub-table, the column-wise domains are selected as the source domain and the row-wise domains are selected as the target domain. ∗ indicates that the backbone is ViT pre-trained on ImageNet-1K.

| ADDA [75]                           | clp inf pnt                                                        | qdr rel Avg   | MIMTFL [26] clp inf pnt qdr                               | rel Avg                    | MDD [97]                                                                           | clp inf pnt qdr         | skt   |               | skt   | Avg   |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | rel skt                                                                                                                                                                                            |                                            |      |                                          |
|-------------------------------------|--------------------------------------------------------------------|---------------|-----------------------------------------------------------|----------------------------|------------------------------------------------------------------------------------|-------------------------|-------|---------------|-------|-------|-------------------------------------------------------------|-------------------------------------|-----|--------|----------------|-------------------------------------|-----------|-----|------|-----------------------------------------|------|----------|----------------------------------------------------------|---------------------------------------|-----------|-----------|------|-----|-------------------------------------------------------------------------------------|----------------------------------------------------------------|--------------------------|--------|----------------------------------------------|------|----------------------------|------|-----------|--------------------------|---------|----------------------------------------|-----------------|-----|-----|---------------------------|------|-----|------------------------------------------------------|-----------|------|------|---------------|----------------------------------------------------------|-----------|---------------------------------------------------------------|------------------|------|-----|------|------|------------------------------------------------------|-----------------------------------------------------------|----|---------------------------------|-----|-----|------|------------------|------|------|-----------|------------------------------------------------------------------------|------|-------------------------------|-----------|----------------|-----------|-----------------------------------------------|------|-----------|------|----------------|------|------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------|------|------------------------------------------|
| clp                                 | - 11.2 24.1 3.2 41.9                                               | 30.7 22.2     | clp                                                       | - 15.1 35.6 10.7 51.5 43.1 | 31.2 clp                                                                           |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | 52.5 42.1 32.4                                                                                                                                                                                     | 6.2                                        |      |                                          |
| inf                                 | 19.1 - 16.4 3.2 26.9                                               | 14.6 16.0     | inf                                                       | 31.0 2.9 48.5 31.0         | - 20.5 40.7 inf 33.0 - 33.8                                                        |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | -    |                                         | 32.1 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    | 2.6                                        | 29.1 |                                          |
| pnt                                 | 31.2 9.5 - 8.4 39.1                                                | 25.4 22.7     | pnt                                                       | 4.2 55.4 36.8 30.2         | 46.2 24.5 28.0 51.2 41.7 32.0                                                      |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 14.7 |                                         | 40.1 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 43.7                                          |      |           |      |                | -    | 20.4                                     |                                                                                                                                                                                                    | 2.8                                        |      | -                                        |
| qdr                                 | 15.7 2.6 5.4 - 9.9                                                 | 11.9 9.1      | qdr                                                       | pnt qdr                    | 10.8                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 18.4                                          |      |           |      |                |      | 3.0                                      | 12.9 11.8                                                                                                                                                                                          | -                                          |      | 8.1                                      |
|                                     |                                                                    |               | 18.8 3.1 5.0 -                                            |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| rel                                 | 39.5 14.5 29.1 12.1 - 25.7                                         | 24.2          | 16.0 13.8 11.3 rel 48.5 19.0 47.6 5.8 - 39.4 32.1         | rel                        | 33.5                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 52.8                                          |      |           |      |                |      | 21.6                                     | - 41.2                                                                                                                                                                                             | 4.2                                        |      | 47.8                                     |
| skt                                 | 35.3 8.9 25.2 14.9 37.6 - 25.4                                     |               | 51.7 16.5 40.3                                            | skt                        | 35.0                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 54.3                                          |      |           |      |                |      | 17.5                                     | 54.2 -                                                                                                                                                                                             | 5.7                                        |      | 43.1                                     |
| Avg.                                | 28.2 9.3 20.1 8.4 31.1 21.7 19.8                                   | skt           | 12.3 53.5 - 34.9 38.2                                     |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     |                                                                    | Avg.          | 13.7 31.9 7.2                                             | 45.0 32.8 28.1 Avg.        | 28.6                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 40.4                                          |      |           |      |                |      | 16.6                                     | 43.4 32.3                                                                                                                                                                                          | 4.3                                        |      | 34.7                                     |
|                                     |                                                                    |               |                                                           |                            | Avg.                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| CDAN [55]                           | clp inf pnt qdr rel skt                                            | Avg. CDAN+GH  | clp inf pnt qdr rel skt Avg.                              | CDAN+GH++                  |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | rel skt                                                                                                                                                                                            | qdr                                        |      | pnt                                      |
|                                     |                                                                    |               |                                                           | clp inf - 21.4             |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| clp                                 | - 20.4 36.6 9.0 50.7 42.3                                          | 31.8 clp      | 20.9 40.8 11.4 57.2 45.8 35.3                             | clp                        | 35.5                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         | -    |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | 56.3                                                                                                                                                                                               | 9.4                                        |      | 40.3                                     |
|                                     |                                                                    |               |                                                           |                            | 46.4                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| inf                                 | 27.5 - 25.7 1.8 34.7 20.1 22.0                                     | inf           | 31.2 3.7 48.2 25.9 28.0                                   | inf                        | 29.0                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | -    |                                         | 31.2 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 33.4                                          |      |           |      |                |      | -                                        | 48.4 26                                                                                                                                                                                            | 5.7                                        |      | 31.6                                     |
| pnt                                 | 42.6 20.0 - 2.5 55.6 38.5 31.8                                     | pnt           | - 2.8 57.7 39.5 32.9                                      | pnt                        | 34.1                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 20.0 |                                         | 44.3 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 45.1                                          |      |           |      |                |      | 20.2                                     | 58.1                                                                                                                                                                                               | 6.2                                        |      | -                                        |
| qdr                                 | 21.0 4.5 8.1 - 14.3 15.7 12.7                                      | qdr           | - 16.9 16.6 14.6                                          | qdr                        | 40.9 16.7 14.6                                                                     |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 4.9  |                                         | 24.0 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 24.2                                          |      |           |      |                |      | 4.8                                      | 16.2                                                                                                                                                                                               | -                                          |      | 10.3                                     |
|                                     | 34.5                                                               |               |                                                           |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| rel                                 | 51.9 23.3 50.4 5.4 - 41.4                                          | rel           | 10.4 53.6 6.1 - 42.3 36.5                                 | rel                        | 36.6                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 24.3 |                                         | 56.0 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      | 24.5                                     | - 42.8                                                                                                                                                                                             | 6.6                                        |      | 53.4                                     |
| skt                                 |                                                                    |               |                                                           | 55.9 57.4                  |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     | 50.8 20.3 43.0 2.9 50.8 - 33.6                                     | skt           | skt                                                       |                            | 38.7                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 21.6 |                                         | 56.9 |          |                                                          |                                       |           |           | -    |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     | 55.1 |                  |      | 11.7 |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 46.1 | 21.5                                     | 55.9 -                                                                                                                                                                                             | 12.3                                       | 38.3 | 46.3                                     |
| Avg.                                | 38.8 17.7 32.8 4.3 41.2 31.6 27.7                                  | Avg.          | 34.1 30.9 Avg.                                            |                            | 34.4 31.4                                                                          |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 18.3 |                                         | 42.5 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | 47.2                                                                                                                                                                                               | 8.5                                        |      | 36.8                                     |
|                                     |                                                                    |               | 34.6 7.1 47.0                                             | MCD+GH++                   | Avg.                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | inf  |                                         | clp  |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | rel                                                                                                                                                                                                | qdr                                        |      | pnt                                      |
| MCD [70]                            |                                                                    |               |                                                           | 43.2 18.5 clp              | skt                                                                                |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     | clp inf pnt qdr rel skt Avg.                                       | MCD+GH        | pnt qdr rel skt Avg. 18.3 37.2 8.4                        |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| clp                                 | - 14.2 26.1 1.6 45.0 33.8                                          | clp           | 43.6 31.9 clp                                             | inf - 19.6 -               | 32.4                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         | -    |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          | 52.2                                                                                                                                                                                               | 8.5                                        |      | 37.2                                     |
|                                     |                                                                    |               | 52.1 2.8 46.7 29.9                                        |                            | 44.3                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| inf                                 | 24.1 23.6 - 21.2 1.5 36.7 18.0                                     | inf           | 29.3                                                      | inf 33.7 44.0              | 30.0 29.5 40.9                                                                     |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | -    |                                         | 34.0 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 33.3 |                                          | 47.1                                                                                                                                                                                               | 3.1                                        |      | 33.8                                     |
| pnt                                 | 20.2                                                               | pnt           | pnt                                                       |                            | 32.3 13.5                                                                          |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 18.9 |                                         | 44.3 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      | 3.7  |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | -    | 19.4                                     | 53.9                                                                                                                                                                                               | 3.9                                        |      | -                                        |
|                                     | 34.4 14.8 - 1.9 50.5 28.4 26.0 9.3                                 |               | 40.7 32.3 11.8                                            |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| qdr                                 | 15.0 3.0 7.0 - 11.5 10.2                                           | qdr           | 54.1 14.5 13.0 39.3                                       | qdr                        | 12.3                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 3.1  |                                         | 20.2 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      | -    |           |                                                                        |      |                               |           |                |           | 20.9                                          |      |           |      |                | 8.4  | 3.6                                      | 15.1                                                                                                                                                                                               | -                                          |      | 8.6                                      |
| rel                                 |                                                                    |               |                                                           | rel                        | 33.6                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 21.5 |                                         | 51.3 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     | -    |                  |      | 2.4  |           |                                                                        |      |                               |           |                |           | 50.9                                          |      |           |      |                | 51.1 | 22.0                                     | - 39.4                                                                                                                                                                                             | 2.9                                        |      | 50.6                                     |
|                                     | 42.6 19.6 42.6 2.2 - 29.3 27.2                                     | rel skt       | 33.0 35.5                                                 | skt                        |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| skt                                 | 41.2 13.7 27.6 3.8 34.8 - 24.2                                     |               |                                                           |                            | 35.5                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     |                                                                    | Avg.          | 51.1 - 33.3 29.0                                          |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 19.1 |                                         | 54.4 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      | 8.6  |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 44.1 | 19.2                                     | 51.3 -                                                                                                                                                                                             | 7.8                                        |      | 43.8                                     |
| Avg.                                | 31.4 13.1 24.9 2.2 35.7 23.9 21.9                                  | DWL+GH        | 43.7 skt Avg.                                             | 54.5 Avg.                  | 29.2                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 16.2 |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| DWL [85]                            | clp inf pnt qdr rel skt                                            |               | DWL+GH++                                                  |                            | Avg.                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | inf  |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     | Avg.                                                               | 40.8 clp -    | 34.8 5.1 pnt qdr rel 31.3                                 |                            | 41.0 16.7 35.2 5.1 43.9 33.5 clp inf pnt qdr rel skt - 17.4 31.3 11.4 49.2         |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| clp                                 | - 17.0 29.5 9.3 48.5 37.2 28.3                                     | clp           | 10.2 49.3 38.2 29.3 22.5 3.0 38.9 23.7 22.5 5.4 46.7 31.0 | clp inf                    | 38.7 29.6 23.8 22.6                                                                |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 17.3 |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| inf                                 | 22.1                                                               | inf           | pnt                                                       |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | -    |                                         | 24.4 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 24.4                                          |      |           |      |                |      | -                                        | 39.2                                                                                                                                                                                               | 3.3                                        |      | 22.4                                     |
| pnt                                 | 24.2 - 22.1 2.5 38.7 23.0 31.2 14.0 - 3.9 46.4 29.5                | pnt           |                                                           |                            | 26.2                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 14.7 |                                         | 31.8 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 32.2                                          |      |           |      |                |      | 15.1                                     |                                                                                                                                                                                                    |                                            |      | -                                        |
| qdr                                 | 25.0 17.9 4.5 10.1 - 16.4 16.4                                     | qdr           | 25.9 13.6                                                 | qdr                        | 13.6 29.7                                                                          |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 4.6  |                                         | 18.7 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 19.0                                          |      |           |      |                |      | 4.8                                      |                                                                                                                                                                                                    |                                            |      | 10.3                                     |
|                                     | 13.1 43.4 20.5 41.4 5.3 - 33.8                                     |               | 29.4                                                      | rel                        | 5.7 46.9 31.1 - 17.6 16.5 5.6 - 34.7 8.9 45.3                                      |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     |                                                                    | rel           | - 10.3 - 17.9 16.6 41.7 5.4 - 34.5                        |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| rel                                 | 28.9                                                               |               |                                                           |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 21.9 |                                         | 43.6 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           | 44.7                                          |      |           |      |                |      | 21.5                                     |                                                                                                                                                                                                    |                                            |      | 42.0                                     |
| skt                                 | 45.4 15.1 32.1 7.3 44.4 - 28.9                                     | skt           | 8.8 45.7 - 30.1                                           | skt                        | - 30.7 29.0 25.4 skt                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 16.0 |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 33.6 | 17.5                                     |                                                                                                                                                                                                    |                                            |      | 34.9                                     |
| Avg.                                | 32.4 14.2 27.0 5.7 38.9 28.0                                       | Avg.          | 25.1 Avg.                                                 |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     | 24.4                                                               |               | 6.6 39.7 28.8 qdr rel                                     |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 14.9 |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 27.9 | 15.3                                     | 39.6                                                                                                                                                                                               | 7.0                                        |      | 28.2                                     |
|                                     |                                                                    | 46.6          |                                                           | 47.1                       |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| GVB [14]                            | clp inf pnt qdr rel skt Avg.                                       | 33.0 GVB+GH   | Avg. GVB+GH++                                             |                            | Avg.                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | inf  |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | pnt  | inf                                      | rel                                                                                                                                                                                                | qdr                                        |      | pnt                                      |
|                                     |                                                                    | clp clp -     | skt 35.4 clp                                              | 33.5 clp                   | 35.5                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| clp                                 | - 18.4 41.5 11.9 56.0 45.2                                         |               |                                                           |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 19.2 |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      | 12.4 |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 41.7 | 19.2                                     | 57.3                                                                                                                                                                                               | 12.8                                       |      | 42.0                                     |
| inf                                 |                                                                    | 35.4          | 57.4 46.2 49.8 29.8                                       | inf                        | 31.7                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     |                                                                    | inf           | 31.5                                                      | - 36.4 47.9                | 46.1 29.2 43.7 35.2                                                                |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | -    |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      | 3.1  |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                | 39.6 | -                                        | 50.0                                                                                                                                                                                               | 3.5                                        |      | 39.6                                     |
| pnt qdr                             | 34.6 33.5 - 36.4 2.3 47.0 27.2 29.3 46.0 19.0 - 5.1 57.4 41.8 33.9 | pnt           | 43.5 34.9 pnt                                             | qdr                        | 19.4                                                                               |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     | 19.5 |                                         | 47.7 |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      | 19.6                                     | 59.4                                                                                                                                                                                               | 5.6                                        |      | -                                        |
|                                     | 31.2 3.0 14.4 - 22.1 21.6 18.5                                     | qdr           | 58.9 22.7 22.0 18.7                                       | rel                        | 37.5 39.1                                                                          |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| rel                                 | 56.0 22.7 54.0 4.4 - 44.7 36.4                                     | rel           | - 5.1 14.4 - 54.7 4.6 - 44.9 37.1                         |                            |                                                                                    |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| skt                                 | 57.6 18.7 48.5 11.1 55.9 - 38.4                                    |               |                                                           |                            | 32.0 3.3 15.2 - 24.0 22.3 58.4 23.3 55.1 5.3 - 45.3 skt 57.9 19.7 48.6 12.1 57.2 - | 40.1 7.9 49.6 37.3 33.1 |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
|                                     |                                                                    | skt           | - 39.0                                                    |                            | 31.4 3.0 57.7 23.4 57.6 19.5 48.5 12.6 56.9                                        |                         |       |               |       |       |                                                             |                                     |     |        |                |                                     |           |     |      |                                         |      |          |                                                          |                                       |           |           |      |     |                                                                                     |                                                                |                          |        |                                              |      |                            |      |           |                          |         |                                        |                 |     |     |                           |      |     |                                                      |           |      |      |               |                                                          |           |                                                               |                  |      |     |      |      |                                                      |                                                           |    |                                 |     |     |      |                  |      |      |           |                                                                        |      |                               |           |                |           |                                               |      |           |      |                |      |                                          |                                                                                                                                                                                                    |                                            |      |                                          |
| Avg. ∗ inf pnt qdr rel rel skt Avg. | 44.9 16.4 39.0 7.0 47.7 36.1 31.8 9.0 8.4                          | Avg.          |                                                           | Avg.                       | 49.1 37.3 32.8 46.5 17.0                                                           |                         | 46.0  | 16.9 39.8 7.6 |       | ∗ ∗   | SSRT [74] clp inf pnt qdr rel skt Avg. SSRT +GH clp inf pnt | qdr rel skt Avg. SSRT +GH++ clp inf | clp | - 33.8 | 60.2 19.4 75.8 | clp - 35.7 60.9 20.9 75.9 61.0 50.9 | 59.8 49.8 | clp |      | inf 55.9 - 55.1 12.2 68.4 50.2 48.4 inf |      | - 55.7 - | pnt qdr rel 35.8 60.9 22.0 76.0 55.2 11.7 69.0 10.5 71.6 | - 10.0 71.7 56.5 46.2 pnt 61.7 33.2 - | 62.1 30.9 | 11.8 24.5 | 43.3 | pnt | qdr 43.9 11.5 24.2 - 37.7 34.4 30.3 qdr rel 70.3 39.6 66.2 10.4 - 59.7 49.2 rel skt | 71.4 33.6 29.3 58.9 48.4 - 52.1 skt 71.0 34.7 62.9 22.8 73.4 - | 68.2 44.7 46.3 55.2 45.0 | - 73.2 | 65.2 50.4 45.2 Avg. 60.6 30.5 53.9 15.3 65.4 | 52.4 | 24.2 - 37.6 66.0 10.1 21.7 | 13.7 | 53.0 46.3 | 37.1 32.8 62.2 28.2 53.3 | SSRT+GH | 70.6 60.0 clp inf pnt qdr rel skt Avg. | clp inf pnt qdr | rel | skt | 54.0 - skt Avg. SSRT [74] | Avg. | clp | - 33.9 60.1 19.4 75.7 61.0 50.0 clp - 35.1 60.7 21.3 | 75.9 61.1 | 50.8 | Avg. | SSRT+GH++ clp | 55.7 - 54.9 9.3 68.2 44.8 46.6 inf 56.3 - 55.6 11.3 68.5 | 50.2 48.4 | inf pnt 62.0 31.5 - 9.3 71.2 55.5 45.9 pnt 62.3 qdr 42.6 12.6 | 31.7 - 10.5 71.5 | 56.7 | inf | 46.5 | 22.2 | - 37.0 34.3 29.7 qdr 44.0 13.0 23.1 - 37.4 34.6 30.4 | pnt qdr 59.4 48.7 rel 71.0 39.5 66.5 10.7 - 59.9 49.5 rel | -  | 71.0 33.9 63.0 22.9 73.5 - 52.9 | skt | skt |      | 21.9 73.2 - 52.0 | 62.4 |      | 10.3 14.0 | 55.5 - 61.7 28.5 42.5 8.8 69.9 70.6 37.0 66.3 70.7 31.9 60.3 29.4 53.2 | 45.5 | 30.6 53.8 15.3 65.4 52.5 46.4 | 65.1 51.0 | Avg. 60.9 31.5 | 71.7 34.8 | 70.0 71.6 60.5 clp - 56.0 62.3 43.5 70.8 62.6 | 60.9 | 53.7 15.8 | Avg. | 23.4 73.4 65.8 |      | 39.8 35.1 31.1 inf 35.8 - 33.3 13.7 39.8 | skt Avg. 61.6 51.3 49.6 48.2 56.3 46.7 38.9 35.1 30.7 - 61.6 49.6 73.4 - 53.2 65.8 52.8 46.6 rel skt Avg. 76.0 61.8 51.5 69.1 49.8 48.3 71.6 56.4 46.9 38.7 35.3 30.9 - 61.5 49.8 - 53.2 53.0 46.8 | - 10.8 23.2 15.6 qdr 22.9 11.4 10.7 - 10.8 |      | 66.0 62.5 53.8 pnt 60.9 55.2 - 23.3 66.3 |

visualization results further validate the balance learning ability of our GH approach.

coordination role in practical optimization process, which promotes the cooperation of domain alignment task and classification task towards a benign direction as expected.

## 5.2 Convergence Analysis

We present the convergence curves of test error with respect to the number of iterations on tasks of U → Mand Synthetic → Real as shown in Fig. 9. For each subfigure, the blue line represents the test error of different baselines, and the red line represents the test error for baseline+GH (e.g., CDAN+GH). Obviously, compared with baselines, the introduction of GH can further improve the test performance and convergence. This fully indicates that GH plays an active

## 5.3 Confusion Matrix Visualization

Fig. 10 displays the visualizations of confusion matrix for the classifier trained by DWL and DWL+GH. DWL obtains several uncertain predictions with small values while DWL+GH obtains more confident predictions. Comparing with Fig. 10 (a) and (b), the confusing 'Class 1, 2, 7, 8, and 9' are correctly recognized in DWL+GH. From Fig. 10 (c) and (d), the confusing 'Class 8' is corrected in DWL+GH.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-007e2490-83a2-4be9-b402-cc31693be6a3.jpg)

(a) MCD on U → M

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-90052d2b-d60c-4d86-954b-496c98fddf0e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-bb2e7182-3e5d-4302-bae5-3ae36b6de53c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-8bb42a4d-d0e3-44a2-9ff0-ca983ba3debd.jpg)

(b) MCD+GH on U → M

(c) MCD on M → U

(d) MCD+GH on M → U

Fig. 8. The t-SNE visualizations of features generated by MCD and MCD+GH. (a) and (b) are feature visualizations on U → M. (c) and (d) are feature visualizations on M → U. Red and blue points indicate the source and target samples, respectively.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-d928ef63-59af-408b-afc9-695231841063.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-0de0a21c-42d1-4af2-be31-fbc45ad22557.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-901b4cc0-4f20-4ce8-9905-2c7fb01f53c4.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-8ef0854e-beaf-40bb-9153-b01bbfd57630.jpg)

46

44

42

40

38

36

34

32

30

(c) Synthetic

→

Real

(d) Synthetic → Real

Fig. 9. Convergence curves of various baselines and baseline+GH on the test error (%). Clearly, the baselines are improved by GH.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-e6b24506-350e-40bd-8a8b-590db6faff5b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-168ef617-8a53-4cc6-87cd-26b8a9f3c686.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-8d294857-0028-4493-a782-9c0d52f58d3d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-ace3164c-6810-4c39-8259-ca7d4e43333e.jpg)

Fig. 10. Confusion matrix visualization Results before and after applying GH (i.e., DWL vs. DWL+GH) on different classification tasks.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-87e7a378-197b-41e1-b094-8d4794c8fd77.jpg)

Fig. 11. Between-task balance analysis for domain alignment (MMD) [3] and class discrimination (max J(w)) [10] of UDA after gradient harmonization. MCD is used in this experiment (i.e., MCD+GH).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-6a06d22c-9aa7-4fa9-b2bf-71f1aa4ff497.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-e66bb991-b363-4b49-a233-25188b35ea0a.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-35d51e83-98c9-48b6-9e8e-cc6455869896.jpg)

It implies that optimization conflict during training deteriorates the discriminability in the target domain. Our method preserves the feature discriminability of target samples in the course of harmonic training. DWL+GH improves the between-task balance by coordinating the explicit external task conflict and the implicit internal optimization conflict.

## 5.4 Balance Analysis of GH-based UDA

Fig. 11 shows MMD distance [3] and the max J(W) [10] values based on the feature representation learned by MCD+GH. The left vertical axis corresponding to the red curve represents the MMD distance used to measure the

Fig. 12. Inner product distributions (histogram) of the two task-specific gradients before and after GH on U → M. Clearly, the between-task gradient conflict is eliminated after harmonization.

TABLE 6 Analyses about the Training Speed (s/epoch) and Classification Accuracy ( % ) before and after applying GH.

| Task   | MCD     | MCD   | MCD+GH   | MCD+GH   | Discrepancy   | Discrepancy   |
|--------|---------|-------|----------|----------|---------------|---------------|
| Task   | Times   | Acc.  | Times    | Acc.     | △ Times       | △ Acc.        |
| M → U  | 0.78540 | 94.2  | 0.92794  | 96.7     | 0.14254       | 2.5           |
| U → M  | 0.89765 | 94.1  | 0.99574  | 96.8     | 0.09809       | 2.7           |

alignment degree across domains. The smaller value means the better domain alignment. The right vertical axis corresponding to the blue curve represents the degree of discrim-

Fig. 13. Recognition accuracy ( % ) on Digits for different gradient harmonization strategies. ' -g 1 &amp; g 2 ' represents multiplying a minus sign to g 1 and keep g 2 unchanged. ' g 1 &amp; -g 2 ' represents multiplying a minus sign to g 2 and keep g 1 unchanged.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-bb145733-8f46-4d27-aff2-ab15444011b6.jpg)

inability based on Linear Discriminant Analysis (LDA) [24]. The larger max J(W) implies better discriminability, which directly affects the classification result of samples. Fig. 11(a) and (b) reflect the degree of domain alignment and discriminability on task of M → U and U → M, respectively. From Fig. 11, MCD+GH has a smaller MMD distance and a larger max J(W) value during training. This shows that GH facilitates the coordinated optimality of alignment and classification tasks while maintaining their task-specific optimality. Enhanced and balanced domain-invariant and class-discriminative feature representations can be obtained.

## 5.5 Training Speed and Accuracy

In order to observe the efficiency of the proposed equivalent model more clearly, we present the training speed and classification accuracy before and after applying GH. As shown in Table 6, for tasks M → U and U → Mon digits datasets, the training speed of MCD+GH is 0.14254 s/epoch and 0.09809 s/epoch longer than MCD, respectively. In other words, MCD+GH takes less training time than MCD in training process, but can get 2.5%/2.7% classification accuracy gain. The computational cost of employing GH is quite low, and thus GH is a powerful and efficient auxiliary tool to facilitate those popular domain adaptation baselines towards more outstanding classification performance.

## 5.6 Gradient Inner Product Visualization

Fig. 12 presents inner product distributions of two gradients between domain alignment and classification tasks before and after applying Gradient Harmonization for MCD. From Fig. 12 (a), we observe the acute and obtuse angles between gradients of the two tasks before coordination. The obtuse angles account for about 40% of the total number, which exhibits the identical property as Fig. 2. In other words, there exists between-task conflict in the model optimization process but paid less attention. After Gradient Harmonization, as shown in Fig. 12 (b), the inner products of the two gradients are all positive. That is, the gradient angles between the two tasks are coordinated into acute angles. The proposed GH avoids the optimization conflict by separately adjusting the gradients of the two tasks to achieve the purpose of optimal coordination. Experimental results fully illustrate the effectiveness of the proposed GH.

## 5.7 Rationality and Comparison to Other Alternatives

The proposed GH/GH++ aims at altering the gradient angle between two different tasks from an obtuse angle

Fig. 14. Sensitivity analysis of the parameter λ in GH++. MCD and GVB are tested as UDA baselines.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00288v1-figure-ff99d48b-eb73-42a9-bf10-0030637c0a4f.jpg)

to an acute/vertical angle. Whether the same effect can be achieved with other intuitive gradient correction alternatives or not? For instance, one might consider convert an obtuse angle into an acute angle by simply applying a negative sign to one of the two gradients, i.e., transforming either g 1 into -g 1 or g 2 into -g 2 . As depicted in Fig. 13, it is apparent that the results reveal a significant decline in classification accuracy when using the -g 1 &amp; g 2 and g 1 &amp; -g 2 methods in contrast to the proposed approaches. While applying a negative sign can indeed transit from an obtuse angle to an acute angle and force the two gradients to be positively correlated, it cannot replicate the excellent classification performance achieved by the proposed GH/GH++.

The rationale behind this observation is that directly applying a negative sign to one of the gradients alters the gradient towards the opposite direction of the original optimal gradient descent direction. This significantly undermines the primary objective of the original task. In other words, the performance deteriorates sharply when the gradient harmonization direction significantly deviates from the original direction. GH/GH++ reasonably adjusts the two gradients to achieve the purpose of optimization coordination on the premise of keeping the harmonic gradient as close as possible to the original gradient descent direction. Therefore, the proposed approaches do not break the taskspecific optimality, but pursues a better cooperation, which verifies the rationality of the proposed GH/GH++.

## 5.8 Parameter Sensitivity Analysis

To investigate the effect of the parameter λ in GH++, we conduct experiments on three tasks (i.e., M → U, U → M and S → M) based on MCD and GVB by varying λ ∈ { 0 0 1 , . , 0 3 . , 0 5 . , 0 7 . , 0 9 . , 1 } . The results are presented in Fig. 14. We can observe GH++ is little sensitive to the scale variety of λ , which indicates that GH++ is robust across different baselines and tasks. Besides, we observe that when λ = 0 5 . , models generally achieve the best performance. In other words, when the gradient deviation of the two tasks is relieved, the performance can be largely improved.

## 5.9 Scalability to Other Multi-task Problems

To further demonstrate the universality and scalability of the proposed approaches, we evaluate GH/GH++ in the object detection and multi-modal interactive retrieval fields, which also involves optimization of multiple objectives.

Dataset. We select widely used benchmarks, i.e., PASCAL VOC 2007 [22] and CSS [80] for object detection and multi-modal interactive retrieval, respectively. PASCAL

TABLE 7 Object detection on PASCAL VOC 2007 test set.

| Method    |   AP |   AP 50 |   AP 60 |   AP 70 |   AP 80 |   AP 90 |
|-----------|------|---------|---------|---------|---------|---------|
| DSSD [23] | 52.4 |    80   |    75.2 |    65   |    46.3 |    16.3 |
| DSSD+GH   | 53   |    80.1 |    75.5 |    65.5 |    47.2 |    17.3 |
| DSSD+GH++ | 53.1 |    80.4 |    75.8 |    65.4 |    47.6 |    17.3 |

TABLE 8 Adversarial training (AT) for multi-modal interactive retrieval on CSS.

| Method       | R@1   | R@1   | R@5   | R@5   | R@10   | R@10   | R@50   | R@50   | R@100   | R@100   |
|--------------|-------|-------|-------|-------|--------|--------|--------|--------|---------|---------|
| Method       | clean | Adv.  | clean | Adv.  | clean  | Adv.   | clean  | Adv.   | clean   | Adv.    |
| TIRG         | 78.8  | 1.8   | 94.9  | 5.9   | 97.3   | 8.6    | 99.1   | 20.2   | 99.5    | 27.9    |
| AT-TIRG [35] | 80.0  | 50.4  | 96.2  | 77.1  | 98.1   | 84.8   | 99.6   | 95.9   | 99.7    | 97.9    |
| AT-TIRG+GH   | 80.6  | 53.8  | 96.3  | 80.0  | 97.9   | 87.0   | 99.5   | 96.4   | 99.8    | 98.2    |
| AT-TIRG+GH++ | 81.0  | 55.6  | 96.4  | 82.0  | 98.2   | 88.7   | 99.7   | 97.0   | 99.9    | 98.5    |

VOC 2007 consists of about 5K trainval images and 5K test images over 20 object categories. CSS consists of 38K synthesized images with different colors, shapes and sizes. It contains about 19K training image-text pairs and 19K testing image-text pairs, respectively.

Implementation Details. For object detection, we use the DSSD [23] model with 321 × 321 inputs as the baseline. We follow the same experimental settings and protocol as [81] and adopt Average Precision (averaged AP at IoUs from 0.5 to 0.9) to measure performance. For multi-modal interactive retrieval, we use TIRG [80] and the adversarial training version(AT-TIRG) follows the paper [35] as the baseline. For evaluation, we follow the same protocols as [34], [35], [80] and use retrieval accuracy R@N as our evaluation metric, which computes the percentage of test queries where at least one target or correctly matched image is within the top N retrieved images. Note that we exactly follow the original experimental setups of the baselines. In other words, DSSD+GH/GH++ and AT-TIRG+GH/GH++ indicate that GH/GH++ are inserted directly into the DSSD and AT-TIRG, respectively, without any change in either the backbone or hyper-parameters.

Results on Object Detection. Object detection models [23], [81] usually consider two tasks: classification and regression. In this section, we use the proposed GH/GH++ to mitigate the conflict between these two tasks. Table 7 presents the Average Precision (AP) on the PASCAL VOC 2007 test set. We can observe that GH/GH++ can improve the AP with different bounding boxes, which verifies the effectiveness of the proposed approaches in object detection.

Results on Multi-modal Interactive Retrieval. Recently, [35] introduces adversarial training [7], [86] into multimodal interactive retrieval model (abbreviated as AT-TIRG) to improve model robustness, which aims to train a generalized and robust model suitable for both clean samples and adversarial attack samples. During adversarial training, there is conflict in training the clean samples and adversarial attack samples, which results in the limitation of the final performance. In this section, we try to adopt the proposed approaches to solve this problem. The experimental results are provided in Table 8. Comparing AT-TIRG+GH with ATTIRG , we can see the proposed GH can enhance the AT. For the adversarial attack samples, AT-TIRG+GH outperforms the AT-TIRG by 3.4%, 2.9%, and 2.2% on R@1, R@5, and

R@10, respectively. AT-TIRG+GH++ outperforms the ATTIRG by 5.2%, 4.9%, and 3.9% on R@1, R@5, and R@10, respectively. The performance of the clean samples is also improved in most cases. Note that the performance of clean samples is usually close to 100%, so there is little space for improvement.

## 6 CONCLUSION

In this paper, we pay attention to the optimization conflict (i.e., imbalance or incoordination) problem between different tasks (i.e., the alignment task and the classification task) in alignment-based unsupervised domain adaptation models. To mitigate this problem, we propose two simple yet efficient Gradient Harmonization approaches, including GH and GH++, which take measures to de-conflict between the gradients of both tasks in optimization. Besides, to facilitate the harmonization during adaptation, we derive the equivalent but more efficient model of UDA with GH/GH++ , which becomes a dynamically reweighted loss function of most existing unsupervised domain adaptation models. Further, the essence and insights of the proposed approaches are provided to indicate its rationality. Exhaustive experiments and model analyses demonstrate that the proposed approaches significantly improve the existing UDA models and contribute to achieving state-of-the-art results. In addition, we have verified that the proposed approaches can be adapted to other problems and areas, such as object detection and multi-modal retrieval, to de-conflict between the gradients of any two tasks in optimization and improve model performance.

## REFERENCES

- [1] S. Azarbarzin and F. Afsari, 'Robust two stage unsupervised metric learning for domain adaptation,' in ICCKE , 2018, pp. 52-57.
- [2] Q. Bi, J. Li, L. Shang, X. Jiang, Q. Liu, and H. Yang, 'Mtrec: Multitask learning over bert for news recommendation,' in Findings of the Association for Computational Linguistics: ACL 2022 , 2022, pp. 2663-2669.
- [3] K. Borgwardt, A. Gretton, M. Rasch, H.-P. Kriegel, B. Schlkopf, and A. Smola, 'Integrating structured biological data by kernel maximum mean discrepancy,' Bioinformatics , vol. 22, pp. 49-57, 08 2006.
- [4] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, 'Domain separation networks,' in NIPS , vol. 29, 2016, pp. 343-351.
- [5] S. Boyd and L. Vandenberghe, Convex Optimization . United States of America by Cambridge University Press, New York, 2004.
- [6] R. Caruana, 'Multitask learning: A knowledge-based source of inductive bias,' in ICML , 1993, pp. 41-48.
- [7] A. Chaturvedi and U. Garain, 'Mimic and fool: A task-agnostic adversarial attack,' IEEE transactions on neural networks and learning systems , vol. 32, no. 4, pp. 1801-1808, 2020.
- [8] M. Chen, S. Zhao, H. Liu, and D. Cai, 'Adversarial-learned loss for domain adaptation,' in AAAI , vol. 34, no. 04, 2020, pp. 3521-3528.
- [9] Q. Chen, Y. Liu, Z. Wang, I. Wassell, and K. Chetty, 'Re-weighted adversarial adaptation network for unsupervised domain adaptation,' in CVPR , 2018, pp. 7976-7985.
- [10] X. Chen, S. Wang, M. Long, and J. Wang, 'Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation,' in ICML , 2019, pp. 1081-1090.
- [11] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, 'Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,' in ICML , 2018, pp. 794-803.
- [12] Z. Chen, J. Ngiam, Y. Huang, T. Luong, H. Kretzschmar, Y. Chai, and D. Anguelov, 'Just pick a sign: Optimizing deep multitask models with gradient sign dropout,' in NIPS , vol. 33, 2020, pp. 2039-2050.

- [13] S. Cui, S. Wang, J. Zhuo, L. Li, Q. Huang, and Q. Tian, 'Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations,' in CVPR , June 2020, pp. 39413950.
- [14] S. Cui, S. Wang, J. Zhuo, C. Su, Q. Huang, and Q. Tian, 'Gradually vanishing bridge for adversarial domain adaptation,' in CVPR , 2020, pp. 12 455-12 464.
- [15] Q. Dai, X.-M. Wu, J. Xiao, X. Shen, and D. Wang, 'Graph transfer learning via adversarial domain adaptation with graph convolution,' IEEE Transactions on Knowledge and Data Engineering , pp. 114, 2022.
- [16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' 2009, pp. 248-255.
- [17] Z. Deng, Y. Luo, and J. Zhu, 'Cluster alignment with a teacher for unsupervised domain adaptation,' in ICCV , 2019, pp. 9943-9952.
- [18] J.-A. D´ esid´ri, e 'Multiple-gradient descent algorithm (mgda) for multiobjective optimization,' Comptes Rendus Mathematique , vol. 350, no. 5-6, pp. 313-318, 2012.
- [19] C. Doersch and A. Zisserman, 'Multi-task selfsupervised visual learning,' in ICCV , 2017, pp. 2051-2060.
- [20] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, 'Decaf: A deep convolutional activation feature for generic visual recognition,' in ICML , 2014, pp. 647-655.
- [21] Z. Du, J. Li, H. Su, L. Zhu, and K. Lu, 'Cross-domain gradient discrepancy minimization for unsupervised domain adaptation,' in CVPR , 2021, pp. 3937-3946.
- [22] M. Everingham, 'The pascal visual object classes challenge, (voc2007) results,' http:// pascallin. ecs. soton. ac. uk/ challenges /VOC /voc2007 /index. html. , 2007.
- [23] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, 'Dssd: Deconvolutional single shot detector,' arXiv:1701.06659 , pp. 1-12, 2017.
- [24] K. Fukunaga, Introduction to statistical pattern recognition , 1990.
- [25] Y. Ganin and V. Lempitsky, 'Unsupervised domain adaptation by backpropagation,' in ICML , 2015, pp. 1180-1189.
- [26] J. Gao, Y. Hua, G. Hu, C. Wang, and N. M. Robertson, 'Reducing distributional uncertainty by mutual information maximisation and transferable feature learning,' in ECCV , 2020, pp. 587-605.
- [27] Z. Gao, S. Zhang, K. Huang, Q. Wang, and C. Zhong, 'Gradient distribution alignment certificates better adversarial domain adaptation,' in ICCV , 2021, pp. 8937-8946.
- [28] B. Geng, D. Tao, and C. Xu, 'Daml: Domain adaptation metric learning,' IEEE Transactions on Image Processing , vol. 20, no. 10, pp. 2980-2989, 2011.
- [29] M. Ghifary, W. B. Kleijn, M. Zhang, D. Balduzzi, and L. Wen, 'Deep reconstruction-classification networks for unsupervised domain adaptation.' in ECCV , 2016, pp. 597-613.
- [30] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in NIPS , 2014, pp. 2672-2680.
- [31] X. Gu, J. Sun, and Z. Xu, 'Spherical space domain adaptation with robust pseudo-label loss,' in CVPR , 2020, pp. 9101-9110.
- [32] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in CVPR , June 2016, pp. 770-778.
- [33] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, 'Cycada: Cycle-consistent adversarial domain adaptation,' in ICML , 2018, pp. 1994-2003.
- [34] F. Huang and L. Zhang, 'Language guided local infiltration for interactive image retrieval,' in CVPR , 2023, pp. 6103-6112.
- [35] F. Huang, L. Zhang, Y. Zhou, and X. Gao, 'Adversarial and isotropic gradient augmentation for image retrieval with text feedback,' IEEE Transactions on Multimedia , vol. 25, pp. 7415-7427, 2023.
- [36] J. J. Hull, 'A database for handwritten text recognition research,' IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 550-554, 1994.
- [37] W. Jiang, H. Gao, W. Lu, W. Liu, F.-L. Chung, and H. Huang, 'Stacked robust adaptively regularized auto-regressions for domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , vol. 31, no. 3, pp. 561-574, 2018.
- [38] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann, 'Contrastive adaptation network for unsupervised domain adaptation,' in CVPR , 2019, pp. 4893-4902.
- [39] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE , vol. 86, no. 11, pp. 2278-2324, 1998.
- [40] C.-Y. Lee, T. Batra, M. H. Baig, and D. Ulbricht, 'Sliced wasserstein discrepancy for unsupervised domain adaptation,' in CVPR , 2019, pp. 10 285-10 295.
- [41] J. Li, E. Chen, Z. Ding, L. Zhu, K. Lu, and H. Shen, 'Maximum density divergence for domain adaptation,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 11, pp. 39183930, 2020.
- [42] K. Li, J. Lu, H. Zuo, and G. Zhang, 'Dynamic classifier alignment for unsupervised multi-source domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , pp. 1-14, 2022.
- [43] M. Li, Y. Zhai, Y. Luo, P. Ge, and C. Ren, 'Enhanced transport distance for unsupervised domain adaptation,' in CVPR , June 2020, pp. 13 936-13 944.
- [44] S. Li, F. Lv, B. Xie, C. H. Liu, J. Liang, and C. Qin, 'Bi-classifier determinacy maximization for unsupervised domain adaptation,' in AAAI , vol. 35, no. 10, 2021, pp. 8455-8464.
- [45] S. Li, W. Ma, J. Zhang, C. H. Liu, J. Liang, and G. Wang, 'Metareweighted regularization for unsupervised domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , pp. 1-14, 2021.
- [46] J. Liang, K. Gong, S. Li, C. H. Liu, H. Li, D. Liu, G. Wang et al. , 'Pareto domain adaptation,' in NIPS , vol. 34, 2021, pp. 12 91712 929.
- [47] J. Liang, D. Hu, and J. Feng, 'Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,' in ICML , 2020, pp. 6028-6039.
- [48] B. Liu, X. Liu, X. Jin, P . Stone, and Q. Liu, 'Conflict-averse gradient descent for multi-task learning,' in NIPS , vol. 34, 2021, pp. 18 87818 890.
- [49] H. Liu, M. Long, J. Wang, and M. Jordan, 'Transferable adversarial training: A general approach to adapting deep classifiers.' in ICML , 2019, pp. 4013-4022.
- [50] H. Liu, M. Shao, Z. Ding, and Y. Fu, 'Structure-preserved unsupervised domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , vol. 31, no. 4, pp. 799-812, 2018.
- [51] L. Liu, Y. Li, Z. Kuang, J. Xue, Y. Chen, W. Yang, Q. Liao, and W. Zhang, 'Towards impartial multi-task learning,' in ICLR , 2021, pp. 1-20.
- [52] M. Liu and O. Tuzel, 'Coupled generative adversarial networks,' in NIPS , vol. 29, 2016, pp. 469-477.
- [53] S. Liu, E. Johns, and A. J. Davison, 'End-to-end multi-task learning with attention,' in CVPR , 2019, pp. 1871-1880.
- [54] M. Long, Y. Cao, J. Wang, and M. I. Jordan, 'Learning transferable features with deep adaptation networks,' in ICML , 2015, pp. 97105.
- [55] M. Long, Z. Cao, J. Wang, and M. I. Jordan, 'Conditional adversarial domain adaptation,' in NIPS , vol. 31, 2018, pp. 1647-1657.
- [56] M. Long, J. Wang, Y. Cao, J. Sun, and S. Y. Philip, 'Deep learning of transferable representation for scalable domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , vol. 28, no. 8, pp. 2027-2040, 2016.
- [57] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, 'Transfer feature learning with joint distribution adaptation,' in ICCV , 2014, pp. 2200-2207.
- [58] M. Long, H. Zhu, J. Wang, and M. I. Jordan, 'Deep transfer learning with joint adaptation networks,' in International conference on machine learning , 2017, pp. 2208-2217.
- [59] Z. Lu, Y. Yang, X. Zhu, C. Liu, Y.-Z. Song, and T. Xiang, 'Stochastic classifiers for unsupervised domain adaptation,' in CVPR , 2020, pp. 9111-9120.
- [60] D. Mahapatra and V. Rajan, 'Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization,' in ICML , 2020, pp. 6597-6607.
- [61] M. Mancini, L. Porzi, S. R. Bulo, B. Caputo, and E. Ricci, 'Inferring latent domains for unsupervised deep domain adaptation,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 2, pp. 485-498, 2021.
- [62] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, 'Cross-stitch networks for multi-task learning,' in CVPR , 2016, pp. 3994-4003.
- [63] J. Na, H. Jung, H. J. Chang, and W. Hwang, 'Fixbi: Bridging domain spaces for unsupervised domain adaptation,' in CVPR , 2021, pp. 1094-1103.
- [64] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng, 'Reading digits in natural images with unsupervised feature learning,' NIPS workshop on deep learning and unsupervised feature learning , pp. 12-17, 2011.

- [65] S. J. Pan and Q. Yang, 'A survey on transfer learning,' IEEE Transactions on Knowledge and Data Engineering , pp. 1345-1359, 2009.
- [66] Y. Pan, T. Yao, Y. Li, Y. Wang, C.-W. Ngo, and T. Mei, 'Transferrable prototypical networks for unsupervised domain adaptation,' in CVPR , June 2019, pp. 2234-2242.
- [67] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, 'Moment matching for multi-source domain adaptation,' in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 1406-1415.
- [68] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko, 'Visda: The visual domain adaptation challenge,' in arXiv:1710.06924 , 2017.
- [69] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, 'Adapting visual category models to new domains,' in ECCV , 2010, pp. 213-226.
- [70] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, 'Maximum classifier discrepancy for unsupervised domain adaptation,' in CVPR , 2018, pp. 3723-3732.
- [71] O. Sener and V. Koltun, 'Multi-task learning as multi-objective optimization,' in NIPS , vol. 31, 2018, pp. 1-12.
- [72] D. Senushkin, N. Patakin, A. Kuznetsov, and A. Konushin, 'Independent component alignment for multi-task learning,' in CVPR , 2023, pp. 20 083-20 093.
- [73] B. Sun and K. Saenko, 'Subspace distribution alignment for unsupervised domain adaptation.' in BMVC , vol. 4, 2015, pp. 1-10.
- [74] T. Sun, C. Lu, T. Zhang, and H. Ling, 'Safe self-refinement for transformer-based domain adaptation,' in CVPR , 2022, pp. 71917200.
- [75] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, 'Adversarial discriminative domain adaptation,' in CVPR , July 2017, pp. 71677176.
- [76] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell, 'Deep domain confusion: Maximizing for domain invariance,' in arXiv:1412.3474 , 2014.
- [77] L. Van der Maaten and G. Hinton, 'Visualizing data using t-sne,' Journal of Machine Learning Research , vol. 9, no. 2605, pp. 2579-2605, 2008.
- [78] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, 'Multi-task learning for dense prediction tasks: A survey,' IEEE transactions on pattern analysis and machine intelligence , vol. 44, no. 7, pp. 3614-3633, 2021.
- [79] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, 'Deep hashing network for unsupervised domain adaptation,' in CVPR , 2017, pp. 5018-5027.
- [80] N. Vo, L. Jiang, C. Sun, K. Murphy, L.-J. Li, L. Fei-Fei, and J. Hays, 'Composing text and image for image retrieval-an empirical odyssey,' in CVPR , 2019, pp. 6439-6448.
- [81] K. Wang and L. Zhang, 'Reconcile prediction consistency for balanced object detection,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 3631-3640.
- [82] X. Wang, L. Li, W. Ye, M. Long, and J. Wang, 'Transferable attention for domain adaptation,' in AAAI , vol. 33, 2019, pp. 53455352.
- [83] G. Wei, C. Lan, W. Zeng, and Z. Chen, 'Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation,' in CVPR , June 2021, pp. 16 643-16 653.
- [84] Y. Wu, D. Inkpen, and A. El-Roby, 'Dual mixup regularized learning for adversarial domain adaptation,' in ECCV , 2020, pp. 540-555.
- [85] N. Xiao and L. Zhang, 'Dynamic weighted learning for unsupervised domain adaptation,' in CVPR , June 2021, pp. 15 242-15 251.
- [86] C. Xie, M. Tan, B. Gong, J. Wang, A. L. Yuille, and Q. V. Le, 'Adversarial examples improve image recognition,' in CVPR , 2020, pp. 819-828.
- [87] M. Xu, J. Zhang, B. Ni, T. Li, C. Wang, Q. Tian, and W. Zhang, 'Adversarial domain adaptation with domain mixup,' in AAAI , 2020, pp. 6502-6509.
- [88] R. Xu, G. Li, J. Yang, and L. Lin, 'Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation,' in ICCV , 2019, pp. 1426-1435.
- [89] Y. Yan, H. Wu, Y. Ye, C. Bi, M. Lu, D. Liu, Q. Wu, and M. K.P. Ng, 'Transferable feature selection for unsupervised domain adaptation,' IEEE Transactions on Knowledge and Data Engineering , pp. 1-15, 2021.
- [90] J. Yang, J. Liu, N. Xu, and J. Huang, 'Tvt: Transferable vision transformer for unsupervised domain adaptation,' in WACV , 2023, pp. 520-530.
- [91] L. Yann, B. Leon, B. Yoshua, and H. Patrick, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE , pp. 2278-2324, 1998.
- [92] S. Ye, K. Wu, M. Zhou, Y. Yang, S. H. Tan, K. Xu, J. Song, C. Bao, and K. Ma, 'Light-weight calibrator: A separable component for unsupervised domain adaptation,' in CVPR , June 2020, pp. 13 736-13 745.
- [93] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, 'Gradient surgery for multi-task learning,' in NIPS , vol. 33, 2020, pp. 5824-5836.
- [94] L. Zhang, J. Fu, S. Wang, D. Zhang, Z. Dong, and C. P. Chen, 'Guide subspace learning for unsupervised domain adaptation,' IEEE transactions on neural networks and learning systems , pp. 33743388, 2019.
- [95] W. Zhang, W. Ouyang, W. Li, and D. Xu, 'Collaborative and adversarial network for unsupervised domain adaptation,' in CVPR , June 2018, pp. 3801-3809.
- [96] Y. Zhang, H. Tang, K. Jia, and M. Tan, 'Domain-symmetric networks for adversarial domain adaptation,' in CVPR , June 2019, pp. 5031-5040.
- [97] Y. Zhang, T. Liu, M. Long, and M. Jordan, 'Bridging theory and algorithm for domain adaptation,' in ICML , 2019, pp. 7404-7413.
Fuxiang Huang Suqi Song Lei Zhang
Gradient Harmonization in Unsupervised Domain Adaptation
Unsupervised domain adaptation (UDA) intends to transfer knowledge from a
labeled source domain to an unlabeled target domain. Many current methods focus
on learning feature representations that are both discriminative for
classification and invariant across domains by simultaneously optimizing domain
alignment and classification tasks. However, these methods often overlook a
crucial challenge: the inherent conflict between these two tasks during
gradient-based optimization. In this paper, we delve into this issue and
introduce two effective solutions known as Gradient Harmonization, including GH
and GH++, to mitigate the conflict between domain alignment and classification
tasks. GH operates by altering the gradient angle between different tasks from
an obtuse angle to an acute angle, thus resolving the conflict and trade-offing
the two tasks in a coordinated manner. Yet, this would cause both tasks to
deviate from their original optimization directions. We thus further propose an
improved version, GH++, which adjusts the gradient angle between tasks from an
obtuse angle to a vertical angle. This not only eliminates the conflict but
also minimizes deviation from the original gradient directions. Finally, for
optimization convenience and efficiency, we evolve the gradient harmonization
strategies into a dynamically weighted loss function using an integral operator
on the harmonized gradient. Notably, GH/GH++ are orthogonal to UDA and can be
seamlessly integrated into most existing UDA models. Theoretical insights and
experimental analyses demonstrate that the proposed approaches not only enhance
popular UDA baselines but also improve recent state-of-the-art models.