text:
\section{Introduction}

This paper considers the unsupervised domain adaptation (UDA) for semantic segmentation. In real-world segmentation tasks, there usually exists a domain gap between the training (source domain) and testing data (target domain), which substantially compromises the segmentation accuracy. Instead of using additional annotated data on the target domain for adaptation, which is notoriously expensive, an alternative way is to adapt the already-learned model through UDA ~\cite{hoffman2018cycada,RN180,RN162}. In another word, we aim to improve the segmentation accuracy on an unlabeled target domain without incurring additional annotation.

A popular pipeline adopted by many state-of-the-art methods~\cite{RN180,RN132,zhang2021prototypical} consists of two training stages, \emph{i.e.}, a warm-up supervised training on the source domain and a sequential self-training on the target domain. Specifically, the first stage trains a warm-up model on the source domain data. For better generalization ability, the warm-up training process is typically assisted with some domain alignment constraints~\cite{RN166,RN135}. Then, the second training stage further adapts the warm-up model to the target domain through self-training~\cite{RN162, zou2019confidence}. The self-training usually uses the warm-up model to assign pseudo labels on the target domain, which are used to re-train (fine-tune) the model.

\begin{figure}
\begin{center}
\includegraphics[width=0.6\textwidth]{images/first_structure.pdf}
\end{center}
\vspace{-20pt}
\caption{The proposed Multiple Fusion Adaptation (MFA) employs a co-learning framework to integrate three information fusions \emph{i.e.}, cross-model fusion, temporal fusion and online-offline pseudo label fusion. The learner (Net A) is co-supervised by the offline pseudo labels, as well as the online labels generated by it co-learner (Net B). To make the online label predictions more stable, MFA smooths the co-learner by temporal average (Mean Net B). Importantly, we design the online pseudo labels to be complementary to the offline pseudo labels, which promotes better fusion effect. In the co-learning framework, Net A and Net B will exchange their role of learner and co-learner. We only present Net A as the learner here for easier understanding.
}
\label{fig: intro}
\end{figure}

This paper proposes a novel and effective Multiple Fusion Adaptation (MFA) method, based on the above-described two-stage UDA pipeline. We employ three basic information fusion to improve the domain adaptation, namely a novel \textbf{online-offline pseudo label fusion}, cross-model fusion and temporal fusion. In MFA, all the three fusion strategies are integrated in a co-learning framework, as illustrated in Figure~\ref{fig: intro}. Each learner is co-supervised by two types of pseudo labels, \emph{i.e.}, the online and the offline pseudo labels. The offline pseudo labels are generated with a popular method ~\cite{RN162}, while the online pseudo labels are generated by the temporal average model of the co-learner. This MFA pipeline has two advantages:

$\bullet$ \emph{The novel online-offline pseudo label fusion.} So far as we know, prior two-stage UDA methods~\cite{RN180,RN132} usually employ the offline pseudo labels. Among the iterations of ``assigning pseudo label'' and ``re-training'', the pseudo labels are updated after several training epochs, yielding the ``offline'' manner. While the offline manner allows additional post-processing and has the advantage of balanced pseudo labels~\cite{RN162}, it is prone to the ignorance of hard and informative samples \cite{RN204}. It is because the offline manner only preserves the most confident predictions among all the target domain data, which are relatively easy. As a remedy, we supplement the offline pseudo labels with online ones, which focus on the relatively hard details (\emph{i.e.}, the informative samples) within each training iteration. Moreover, since the online pseudo labels are generated by the up-to-date model (which has better accuracy than the historical ones), they reduce the exposure to noisy supervision and thus benefit the self-training process.

$\bullet$ \emph{A highly efficient integration of three fusion strategies.} While the cross-model fusion and temporal fusion are quite popular, MFA pays significant efforts to raise the efficiency and effectiveness for integration, and succeeds in injecting all the three strategies into a unified framework. Specifically, MFA employs a co-learning framework consisted of two learners, as illustrated in Figure~\ref{fig: intro}. Each learner (model) in MFA is co-supervised by the offline pseudo labels generated by itself, as well as the online labels generated by its co-learner. To make the online predictions more stable, MFA smooths the co-learner by temporal average. Consequentially, MFA simultaneously enforces information fusion between 1) a model and its co-learner (and vice versa), 2) the up-to-date status and the temporal-averaged status and 3) online and offline pseudo labels. Combining these parallel fusions, MFA suppresses the pseudo label noises in the self-training stage and thus improves the segmentation adaptation.

Equipped with these two advantages, MFA is capable to improve UDA for semantic segmentation. First, the novel online-offline pseudo label fusion enables MFA to retain more informative details for adaptive training. Second, MFA manages a highly efficient integration of the online-offline pseudo label fusion and two commonly-adopted fusions, which further increases the accuracy of the pseudo labels. We evaluate the proposed MFA through extensive experiments. Experimental results show that MFA significantly improves the baseline and achieve performance on par with the state of the art. For example, on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes, MFA achieves $58.2\%$ and $62.5\%$ mIoU, respectively. Moreover, ablation study validates the effectiveness of each component in MFA. The main contributions of this paper are summarized as follows:

\begin{itemize}
\vspace{-2pt}
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
\setlength{\parskip}{0pt}
    \item We propose MFA, an unsupervised semantic segmentation adaptation method based on self-training. MFA efficiently integrates three different information fusion strategies to improve the pseudo-label-based UDA.
    \item Among the three fusion strategies of MFA, the online-offline pseudo label fusion is a novel one specifically designed for adaptive segmentation. The online pseudo labels supplement the self-training with relatively hard and informative samples, which may be easily ignored by the offline pseudo labels.

    \item We conduct extensive experiments to evaluate the proposed MFA. Experimental results show that MFA achieves superior adaptive semantic segmentation and the training cost is relatively low.

\end{itemize}

\section{Related work}

\paragraph{Semantic segmentation adaptation.} We divide the existing UDA semantic segmentation methods into two categories: domain alignment~\cite{hoffman2018cycada,RN180,RN138,RN135} and self-training~\cite{RN162,zou2019confidence,shin2020two,paul2020domain,pan2020unsupervised}, and the existing state-of-art approaches are usually a combination of two methods. The main motivation of domain alignment is to reduce the discrepancy between two domains.
CyCADA~\cite{hoffman2018cycada} uses CycleGan~\cite{zhu2017unpaired} to transfer image style. FDA~\cite{RN180} proposes exchanging the low-frequency component of fourier transform without learning to achieve the same purpose.
In addition, SIM~\cite{RN135} introduces the feature alignment of things and stuff respectively.
In self-training, pseudo label learning~\cite{RN162} is a widely used approach. CBST~\cite{RN162} proposes an iterative pseudo label learning strategy and solve the class imbalance issue by class-independent confidence ranking. The improvement of pseudo label learning is an important research direction. In CRST~\cite{zou2019confidence}, a confidence regularized self-training method is proposed to address the problem of overconfident wrong label.~\cite{shin2020two} presents a two-phase pseudo label densification framework through voting-based and  easy-hard classification based method. In ~\cite{paul2020domain}, weak labels are explored to enhance pseudo label learning. Our work considers suppressing the pseudo label noise through multiple fusion strategies.

\paragraph{Temporal average.} Temporal ensembling~\cite{laine2016temporal} averages the outputs of the network-in-training to increase the prediction accuracy for the unlabelled samples. The mean teacher~\cite{tarvainen2017mean} averages model weights at different training steps to get a teacher model. The teacher model offers supervision signal through consistency constraint on the unlabeled samples. These works show that the temporal average of the deep model is more stable and accurate than the deep model at a single training step.

\paragraph{Learning with noisy labels.}  The information fusion between different models is an effective approach to suppress noises in labels. Co-teaching~\cite{han2018co} cross-trains two networks and let them teach each other given the possibly clean labels by small-loss trick. Co-teaching+~\cite{yu2019does} bridges the "Disagreement" strategy with the Co-teaching to enhance robustness under extremely noisy supervision. In these works, the labels are all available, which is different from the unsupervised domain adaptation problem.

We note that~\cite{RN132,zhang2021prototypical} also consider the noise issue of pseudo labels on semantic segmentation adaptation. ~\cite{RN132} estimates the uncertainty of predictions and reduces the impact of low-confidence samples during pseudo label learning. \cite{zhang2021prototypical} take this issue by exploiting the feature distances from prototypes. Our method tackles the noisy pseudo label problem from a different viewpoint. We use co-learning and integrate multiple fusion strategies to resist the noisy pseudo labels, as well as to retain informative samples. Experimental results show that the proposed MFA marginally surpasses~\cite{RN132, zhang2021prototypical}.

\section{Approach}
The proposed MFA adopts the popular UDA pipeline of two-stage training, \emph{i.e.}, a warm-up training on the source domain and a following self-training on the target domain. We first give a formal description of the two-stage UDA pipeline in Section \ref{sec: pre}.
Based on this pipeline, MFA improves the adaptive segmentation through multiple fusions in the self-training stage, as illustrated in Figure~\ref{image:learn_step}. Basically, MFA uses two independent models (Net A and Net B) to set up a co-learning framework. Both models have the dual role of learner and co-learner. Before we collaboratively fine-tune them through self-training, we combine the two warm-up models to generate offline pseudo labels on the target domain (Section \ref{sec: offline}). During self-training, MFA smooths each model through temporal moving average and gets a corresponding ``mean net'' (\emph{i.e.}, Mean Net A and Mean Net B in Figure~\ref{image:learn_step}).  The function of Mean Net (or the temporal moving average operation) is two-fold. First, temporal moving average stabilizes the update of each mean net , therefore making the online predictions more stable. Second, according to the discovery in semi-supervised learning~\cite{tarvainen2017mean}, temporal moving average benefits from the ensemble of multiple models and thus maintains higher prediction accuracy. Given the current training mini-batch , each mean net predicts a respective set of online pseudo labels (Section \ref{sec: online}). Finally, MFA enforces co-supervision on each model. Specifically, each learner is co-supervised by the offline pseudo labels, as well as the online ones generated by its co-learner (Section \ref{sec: co-supervision}).

\subsection{Preliminaries on Two-stage UDA} \label{sec: pre}
In domain adaptive segmentation, we have two datasets belonging to different domains, \emph{i.e.}, the source domain and the unlabeled target domain. The source domain dataset is denoted as $D_{S}=\left \{ x_{S}^{i}, y_{S}^{i} \right \}_{i=1}^{N_{S}}$, where $x_{S}\in\mathbb{R}^{H \times W \times 3}$ is a color image in source domain, $N_{S}$ is the number of source data and $y_{S} \in \mathbb{R}^{H \times W}$ is the corresponding semantic map. The definition of the target domain dataset $D_{T}=\left \{ x_{T}^{j}, y_{T}^{j} \right \}_{j=1}^{N_{T}}$ is similar, except that $y_{T}$ is unknown. Let $F$ represents a semantic segmentation network, and $\theta$ stands the parameters of $F$. The goal of the UDA problem is to estimate $\theta$ to minimize the prediction error on the unlabeled target domain.

In two-stage UDA, the parameter $\theta$ of the warm up model are first obtained through training on the source domain (\emph{i.e.}, stage 1). Then in the self-training (\emph{i.e.}, stage 2), given the input sample from target domain, the prediction $\hat{y}_{T} = F\left ( x_{T} \mid \theta\right )$ is the predicted class probability map, where $\hat{y}_{T}\in\mathbb{R}^{H \times W \times C}$  and $C$ is the number of classes. And $\mathop{\max}\left(\hat{y}_{T}^{c}\right)\in\mathbb{R}^{H \times W}$ is the prediction confidence map. The one-hot map of pseudo labels is obtained by:%
\begin{equation}
\label{equ:option5}
\hat{P}\left ( x_{T} \mid \theta \right )=\text{one-hot}\left(\mathop{\arg\max}_{c}F\left ( c \mid x_{T}, \theta\right)\right)
\end{equation}%
In the standard self-training strategy, an optional way is to retrain the initialized model multiple times by merging the pseudo label data with the source data, which is applied in~\cite{RN135,RN180}. However, this strategy needs to reinitialize $F$ to start training, which is very time-consuming. Therefore, we choose another way used in~\cite{RN132} as our baseline, \emph{i.e.}, fine-tune the warm-up model on the pseudo labels. The loss function for self-supervision is formulated as follows:%
\begin{equation}
\label{equ:option2}
\mathcal{L}_{self}(x_{T},\theta) =-\mathop{\sum}_{batch}m \cdot \hat{p}_{T} \cdot \log\left(F\left ( x_{T} \mid\theta\right )\right)
\end{equation}%
Where $\hat{p}_{T}$ is obtained by $x_{T}$ and  $\theta$ in Equation~\ref{equ:option5}. And $m \in \mathbb{R}^{H \times W}$ is a binary mask for filtering out the unreliable pseudo labels. Specifically, if $m_{h, w}=1$, then we have $\hat{P}_{T, h, w}$ selected for training. In contrast, if $m_{h, w}=0$, the corresponding pseudo label is regarded as unreliable and thus ignored.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{images/second_structure.pdf}
\caption{\textbf{\textcolor{red}{(a)}} The overall framework of the proposed Multiple Fusion Adaptation (MFA). MFA collaboratively trains two models (Net A and Net B), with two supervision signals (Section \ref{sec: co-supervision}), \emph{i.e.}, the offline and online pseudo labels. The offline pseudo labels are generated by a popular baseline CBST \cite{RN162}, as introduced in Section \ref{sec: offline}. The online pseudo labels for Net A are generated by the temporal average of Net B (mean Net B), and vice versa (Section \ref{sec: online}).

\textbf{\textcolor{red}{(b)}} Some examples of the online and offline pseudo labels.
We highlight some regions with red bounding boxes to draw attention of the complementarity between these labels. For example, in the first column, the offline pseudo labels omit some important details about the pedestrians, and the online pseudo labels make up for this problem. In the second column, the person riding a motorcycle is recalled by online pseudo labels.}
\label{image:learn_step}
\end{figure}
\subsection{Offline Pseudo Label}\label{sec: offline}
In Equation~\ref{equ:option2}, the pseudo labels are typically selected in an offline manner, \emph{i.e.}, they will not be instantly updated during model optimization.
MFA combines two different warm-up models for offline pseudo label prediction $\tilde{y}_{T}$.
Consequentially, the offline pseudo labels benefit from model ensemble. Given the raw predictions, we use the CBST~\cite{RN162} method to select the training samples (pixels) by:%
\begin{equation}
\label{equ:option7}
m =
\begin{cases}
1  & \text{ if }\, c=\mathop{\arg\max}\left ( \tilde{y}_{T}^{c} \right )\, \& \, \mathop{\max}\left(\tilde{y}_{T}^{c}\right)>\tau_{c} \\
0 & otherwise
\end{cases}
\end{equation}%
where $c\in \left [ 0, C\right )$ is the class index, and $\tau_{c}$ is the thresholds for the corresponding class. Following~\cite{RN162}, we set each threshold $\tau_{c}$ to ensure class balance, \emph{i.e.}, all the classes has an identical proportion of the selected pixels.

The offline manner of CBST has the advantage of stabilizing supervision signals and avoiding the gradual dominance of large classes~\cite{RN162}. In MFA, the ensemble of two warm-up models further benefits the accuracy of the pseudo labels. However, the offline manner is prone to the problems of \emph{ignorance of hard samples} \cite{RN204} and \emph{longer exposure to the potential noisy labels}. We thus introduce the online pseudo labels as a supplementary.

\subsection{Online Pseudo Label} \label{sec: online}
\paragraph{Temporal average.} MFA instantly assigns online pseudo labels to images in the current mini-batch. In another word, the online pseudo labels do NOT require a post-processing over the whole target domain samples. To alleviate the potential instability issue, MFA first smooths each model through temporal average to generate mean net, which is formulated as:
\begin{equation}
\label{equ:option3}
\theta_{t}^{mean} = \alpha \theta_{t-1}^{mean}+(1-\alpha)\theta_{t}
\end{equation}%
Where $\alpha$ is a smoothing coefficient hyper-parameter, $\theta$ denotes the model parameters and $t$ is the training step. Given the mean net with parameters $\theta^{mean}$ and an input image $x_T$, MFA generates the corresponding online pseudo labels by Equation~\ref{equ:option5}.

\begin{wrapfigure}{R}{0.55\textwidth}
 \vspace{-9pt}
    \begin{minipage}{0.55\textwidth}
      \begin{algorithm}[H]
        \caption{Online CBST}
        \label{alg:cbst}
        \textbf{Input}: Mean Net $F(\theta_{A})$ and $F(\theta_{B})$, minibatch target data $x_{b}$. \\
        \textbf{Parameter}: Ratio $ \varphi $ of selected pseudo labels.\\
        \textbf{Output}: $LP_{b}^{1}$ and $LP_{b}^{2}$ from $\theta_{A}$ and $\theta_{B}$, respectively.
        \begin{algorithmic}
            \STATE $P_{b}^{1} = F\left ( x_{b} \mid \theta_{A} \right )$
            \STATE $P_{b}^{2} = F\left ( x_{b} \mid \theta_{B} \right )$
            \STATE $LP_{b}^{1} = \arg\max\left ( P_{b}^{1},axis=1\right ) $
            \STATE $LP_{b}^{2} = \arg\max\left ( P_{b}^{2},axis=1\right ) $

            \FOR {i=1\ to\ 2}

            \STATE $MP_{b}^{i} = \max\left ( P_{b}^{i},axis=1\right ) $
            \FOR {c=1\ to\ C}
            \STATE $MP_{c, b}^{i} = MP_{b}^{i}\left [ LP_{b}^{i}==c \right ]$
            \STATE $M_{c}^{i} = sort\left ( MP_{c, b}^{i}, order=descending \right ) $
            \STATE $len_{c}^{i} = length(M_{c}^{i})\times \varphi$
            \STATE $\tau_{c}^{i} = M_{c}^{i}\left [ len_{c}^{i} \right ] $
            \STATE $LP_{b}^{i}\left[ LP_{b}^{i}==c \ \& \ LP_{b}^{i} < \tau_{c}^{i}  \right] = 255$
            \ENDFOR
            \ENDFOR
            \STATE \textbf{return} $LP_{b}^{1}$, $LP_{b}^{2}$
        \end{algorithmic}
      \end{algorithm}
    \end{minipage}
\end{wrapfigure}

\paragraph{Online CBST.} The online pseudo labels require to be filtered as well, so as to remove the labels with relatively low confidence. To this end, we propose a novel Online-CBST.

It is similar to the original CBST, except that it uses the data in current mini-batch (rather than the whole training data) as the reference for label selection. In other words, we set a respective filtering threshold for each class, so that all the classes have equal proportion of pseudo-labeled samples in current mini-batch. In analogy to the original CBST, we design the Online-CBST to have linearly-increasing proportion $\varphi(t) \in \left[\rho_{min}, \rho_{max} \right]$, which is the desired proportion in the $t-$th training step. $\rho_{min}$ and $\rho_{max} $ are the pre-defined minimum and maximum proportions.
Initially, the accuracy of each warm-up model is relatively low. So we use a small $\varphi(0)=\rho_{min}$ to retain the most confident pseudo labels and abandon the others.

Since the online pseudo labels are generated by the up-to-date model, they reduce the exposure to noisy supervision. Moreover, Online CBST is beneficial for recalling the relatively hard details (\emph{i.e.}, the informative samples) within each training iteration. In Section~\ref{sec:Role of Online}, we analyzed this in more detail. Given the online pseudo labels $P(x_T\mid\theta_A^{mean})$ and $P(x_T\mid\theta_B^{mean})$, the Online-CBST correspondingly generates two masks $m_A$ and $m_B$ for selecting the pixels. The pseudo code for generating the online labels is to be accessed in Algorithm~\ref{alg:cbst}.

\subsection{Co-supervision in MFA} \label{sec: co-supervision}

Given both the offline and online pseudo labels, MFA enforces a co-supervision on each learner in Figure~\ref{image:learn_step} (a). We illustrate the loss functions for such co-supervision as follows.

To cooperate with the offline pseudo labels, MFA uses the loss function $\mathcal{L}_{self}$ defined by Equation~\ref{equ:option2} for self-supervision on both Net A and Net B. As for the online pseudo labels, MFA uses $P(x_T\mid\theta_A^{mean})$ (predictions from Net A) for supervising Net B and uses $P(x_T\mid\theta_B^{mean})$ for supervising Net A, yielding the so-called cross-model supervision. The detailed loss functions are formulated as:%
\begin{equation}
\label{equ:option9}
\begin{aligned}
&\mathcal{L}_{cross}(\theta_{A}, \theta_{B}^{mean}) = -\mathop{\sum}_{batch}m_{\text{B}}\cdot P\left ( x_{T} \mid \theta_{B}^{mean} \right )   \cdot \log \left (F\left ( x_{T} \mid\theta_{A}\right )\right ) \\
& \mathcal{L}_{cross}(\theta_{B}, \theta_{A}^{mean}) = -\mathop{\sum}_{batch}m_{\text{A}}\cdot P\left ( x_{T} \mid \theta_{A}^{mean} \right ) \cdot \log \left (F\left ( x_{T} \mid\theta_{B}\right ) \right )
\end{aligned}
\end{equation}%
in which $m_B$ ($m_A$) is the selection-mask for $P(x_T\mid\theta_B^{mean})$ ($P(x_T\mid\theta_A^{mean})$) generated by the proposed Online-CBST.

Besides the co-supervision with online and offline pseudo labels, MFA enforces a respective consistency constraint between each learner and its temporal average. Similar to mean teacher~\cite{tarvainen2017mean}, the consistency loss is the expected distance between the prediction of the model and the prediction of the temporal average model, which is formulated as:%
\begin{equation}
\label{equ:option4}
\mathcal{L}_{cst}(\theta, \theta^{mean}) = \mathbb{E}_{x}\left [ \left \| F\left ( x_{T} \mid\theta^{mean}\right ) - F\left ( x_{T} \mid \theta\right ) \right \| \right ]
\end{equation}%
In summary, MFA sums up all the losses to collaboratively train Net A and Net B by:%
\begin{equation}
\label{equ:option10}
\begin{aligned}
\mathcal{L}_{all} = &\mathcal{L}_{self}\left(x_{T},\theta_{A}\right)+\mathcal{L}_{self}(x_{T},\theta_{B}) \\
 &+ \lambda_{cst} \left(\mathcal{L}_{cst}\left(\theta_{A},\theta_{A}^{mean}\right)+\mathcal{L}_{cst}\left(\theta_{B},\theta_{B}^{mean}\right)\right) \\
 &+  \lambda_{cross}\left(\mathcal{L}_{cross}\left(\theta_{A}, \theta_{B}^{mean}\right)+\mathcal{L}_{cross}\left(\theta_{B}, \theta_{A}^{mean}\right)\right),
\end{aligned}
\end{equation}%
in which $\lambda_{cst}$  and $\lambda_{cross}$  are the weighting factors for $\mathcal{L}_{cst}$  and $\mathcal{L}_{cross}$, respectively.

\section{Experiments}
\subsection{Datasets}
We evaluate the proposed MFA under two widely adopted cross-domain segmentation settings, \emph{i.e.}, GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes.
GTA5 and SYNTHIA are both synthetic datasets. The GTA5~\cite{richter2016playing} dataset consists of 24,966 synthesized images of resolution $1914 \times 1052$. Same as existing works, we evaluate our method on 19 common categories shared by GTA5 and Cityscapes. The SYNTHIA~\cite{ros2016synthia} dataset has 9,400 synthesized images of resolution $1280 \times 720$ with fine annotations. Following~\cite{RN135,RN180}, we report the per-class IoU and mIoU on the 13 common categories shared by SYNTHIA and Cityscapes.

Cityscapes is a real-world semantic segmentation dataset~\cite{cordts2016cityscapes}, which consists of $5,000$ images of resolution $2048 \times 1024$ with pixel-level annotations.
It is split into a training set, validation set and test set with $2,975$, $500$ and $1,525$ images, respectively. In line with the standard evaluation setting, we use the $2,975$ training images (without the ground-truth labels) as target domain images, and then evaluate the domain adaptive segmentation accuracy on the validation set.

\subsection{Implementation Details} \label{section:architecture}
Following \cite{RN135,RN180}, we use DeepLabV2~\cite{chen2017deeplab} based on ResNet101 \cite{he2016deep} as the backbone model. We recall that MFA is based on the two-stage UDA pipeline and requires warm-up training. To promote the divergence between the initial status of two learners, we adopt two state-of-the-art domain alignment methods proposed by FDA~\cite{RN180} and SIM~\cite{RN135}.
These two methods serves as the strong baseline for MFA. That being said, we will show that MFA achieves significant improvement over these (\emph{e.g.}, +10.1\% mIoU on GTA5-to-Cityscapes), which results in the state-of-the-art performance. Moreover, MFA is compatible to any warm-up models and is potential to benefit from the future progress in domain alignment.

We use SGD optimization strategy with momentum 0.9. We initialize the learning rate to $2e^{-4}$, and adjust it according to the poly learning rate scheduler with a power of 0.9. As for the hyper-parameters, Equation~\ref{equ:option3} has $\alpha = 0.99$ for temporal average, and Online-CBST has $\rho_{min}=0.2$, $\rho_{max}=0.7$. Moreover, we set $\lambda_{cst}=1.0$, $\lambda_{cross}=0.5$ for Equation~\ref{equ:option10}.

\begin{table}
\begin{center}
\resizebox{\textwidth}{20mm}{
\begin{tabular}{|l|cccccccccccccccccccc|}
\hline
Method & \rotatebox{90}{road} & \rotatebox{90}{sdwk} & \rotatebox{90}{bldng} & \rotatebox{90}{wall} & \rotatebox{90}{fence} & \rotatebox{90}{pole} & \rotatebox{90}{light} & \rotatebox{90}{sign} & \rotatebox{90}{veg} & \rotatebox{90}{trrn} & \rotatebox{90}{sky} & \rotatebox{90}{psn} & \rotatebox{90}{rider} & \rotatebox{90}{car} & \rotatebox{90}{truck} & \rotatebox{90}{bus} & \rotatebox{90}{train} & \rotatebox{90}{moto} & \rotatebox{90}{bike} & \rotatebox{90}{mIoU} \\
\hline\hline
AdaStruct~\cite{RN133} & 86.5 & 25.9 & 79.8 & 22.1 & 20.0 & 23.6 & 33.1 & 21.8 & 81.8 & 25.9 & 75.9 & 57.3 & 26.2 & 76.3 & 29.8 & 32.1 & 7.2 & 29.5 & 32.5 & 41.4     \\
Cycada~\cite{hoffman2018cycada} & 86.7 & 35.6 & 80.1 & 19.8 & 17.5 & 38.0 & 39.9 & 41.5 & 82.7 & 27.9 & 73.6 & 64.9 & 19.0 & 65.0 & 12.0 & 28.6 & 4.5 & 31.1 & 42.0 & 42.7 \\
WSDA~\cite{paul2020domain} & 91.6 & 47.4 & 84.0 & 30.4 & 28.3 & 31.4 & 37.4 & 35.4 & 83.9 & 38.3 & 83.9 & 61.2 & 28.2 & 83.7 & 28.8 & 41.3 & 8.8 & 24.7 & 46.4 & 48.2 \\
BDA~\cite{RN166} & 91.0 & 44.7 & 84.2 & 34.6 & 27.6 & 30.2 & 36.0 & 36.0 & 85.0 & 43.6 & 83.0 & 58.6 & 31.6 & 83.3 & 35.3 & 49.7 & 3.3 & 28.8 & 35.6 & 48.5 \\
SIM~\cite{RN135} & 90.6 & 44.7 & 84.8 & 34.3 & 28.7 & 31.6 & 35.0 & 37.6 & 84.7 & 43.3 & 85.3 & 57.0 & 31.5 & 83.8 & \textbf{42.6} & 48.5 & 1.9 & 30.4 & 39.0 & 49.2     \\
Seg-U~\cite{RN132} & 90.4 & 31.2 & 85.1 & 36.9 & 25.6 & 37.5 & \textbf{48.8} & \textbf{48.5} & 85.3 & 34.8 & 81.1 & 64.4 & 36.8 & 86.3 & 34.9 & 52.2 & 1.7 & 29.0 & 44.6 & 50.3\\
FDA~\cite{RN180} & 92.5  & 53.3  & 82.4  & 26.5  & 27.6  & 36.4  & 40.6  & 38.9  & 82.3  & 39.8  & 78.0  & 62.6  & 34.4  & 84.9  & 34.1  & 53.1  & 16.9  & 27.7  & 46.4  & 50.5\\
TPLD~\cite{shin2020two} & 94.2 & 60.5 & 82.8 & 36.6 & 16.6 & 39.3 & 29.0 & 25.5 & 85.6 & \textbf{44.9} & 84.4 & 60.6 & 27.4 & 84.1 & 37.0 & 47.0 & \textbf{31.2} & 36.1 & 50.3 & 51.2 \\
ProDA~\cite{zhang2021prototypical} & 91.5 & 52.4 & 82.9 & \textbf{42.0} & \textbf{35.7} & 40.0 & 44.4 & 43.3 & \textbf{87.0} & 43.8 & 79.5 & 66.5 & 31.4 & 86.7 & 41.1 & 52.5 & 0.0 & 45.4 & 53.8 & 53.7 \\
MFA(ours)  & \textbf{94.5} & \textbf{61.1} & \textbf{87.6} & 41.4 & 35.4 & \textbf{41.2} & 47.1 & 45.7 & 86.6 & 36.6 & \textbf{87.0} & \textbf{70.1} & \textbf{38.3} & \textbf{87.2} & 39.5 & \textbf{54.7} & 0.3 & \textbf{45.4} & \textbf{57.7} & \textbf{55.7}     \\
\hline
ProDA*~\cite{zhang2021prototypical} & 87.8 & 56.0 & 79.7 & 46.3 & 44.8 & 45.6 & 53.5 & 53.5 & 88.6 & 45.2 & 82.1 & 70.7 & 39.2 & 88.8 & 45.5 & 59.4 & 1.0 & 48.9 & 56.4 & 57.5 \\
MFA*(ours) & 93.5 & 61.6 & 87.0 & 49.1 & 41.3 & 46.1 & 53.5 & 53.9 & 88.2 & 42.1 & 85.8 & 71.5 & 37.9 & 88.8 & 40.1 & 54.7 & 0.0 & 48.2 & 62.8 & \textbf{58.2} \\

\hline
\end{tabular}}
\end{center}
\caption{ Results on GTA5-to-Cityscapes. MFA surpasses all the competing methods. For a fair comparison, ``*'' indicates additional distillation stage is used, which is proposed in~\cite{zhang2021prototypical}. }
\label{tab:result_gta5}
\vspace{-0.2cm}
\end{table}

\subsection{The Effectiveness of MFA}
Table~\ref{tab:result_gta5} compares MFA against several state-of-the-art UDA methods on the GTA5-to-Cityscapes benchmark, from which we draw two observations.
First, comparing MFA against all the competing two stage methods, we find that MFA surpasses all the competing methods by a large margin.
For example, it achieves $55.7\%$ mIoU, which is higher than the strongest competitor ProDA by $+2.0\%$. We achieve the best scores on most categories (11 in 19). Second, under the fair comparison, MFA* presents $58.2\%$ mIoU, with additional model distillation stage~\cite{zhang2021prototypical}. In line with ~\cite{zhang2021prototypical}, we use the SimCLRv2~\cite{chen2020big} pretrained weights and same distillation strategy.

We also compare MFA with competing methods on the SYNTHIA-to-Cityscapes benchmark in the Table~\ref{tab:result_syn} and draw consistent observations as on GTA5-to-Cityscapes. MFA achieves higher mIoU than prior state-of-the-art methods. We report $58.7\%$ mIoU (after self-training stage) and $62.5\%$ mIoU (after distillation stage) on this benchmark.

\subsection{The Efficiency of MFA}
The basic self-training strategy needs iteration of ``assigning pseudo label'' and ``re-training'', which is adopted by~\cite{RN162,RN180}. Benefit from the online-offline fusion, MFA converges faster without multiple iterations and reinitialization. The training in MFA lasts 65 epochs, and is more efficient than other methods~\cite{RN132,RN180,zhang2021prototypical,RN162}. We note that the warm-up models in FDA~\cite{RN180} and the proposed MFA achieve close performance, but MFA achieves 55.7 mIoU after self-training stage (compared to 50.5 in FDA after two round self-training stage). We thus infer that the superiority of MFA is due to the well-engineered self-training with multiple information fusion.

\subsection{The Benefit of Online Pseudo Labels} \label{sec:Role of Online}
We visualize some examples of online and offline pseudo labels in Figure~\ref{image:learn_step} (b). It is clearly observed that online pseudo labels are complementary to the offline ones and focus on the relatively hard details. In the first column, the offline pseudo labels omit the pedestrians, which look small in the image. In contrast, the online pseudo labels succeed in pointing the existence of these pedestrians. In the second column, the person riding a motorcycle is omitted by the offline pseudo labels and is recalled by online pseudo labels. We owe this to two reasons. First, the mean net is updated in time, which is conducive to producing more high-quality pseudo labels along with self-training. Second, since the proposed online CBST focuses on current batch of samples, the relatively difficult samples are more likely to be recalled in the confidence ranking.

\begin{table}
\begin{center}
\resizebox{\textwidth}{20mm}{
\begin{tabular}{|l|cccccccccccccc|}
\hline
Method & \rotatebox{90}{road} & \rotatebox{90}{sdwk} & \rotatebox{90}{bldng} & \rotatebox{90}{light} & \rotatebox{90}{sign} & \rotatebox{90}{veg} & \rotatebox{90}{sky} & \rotatebox{90}{psn} & \rotatebox{90}{rider} & \rotatebox{90}{car} & \rotatebox{90}{bus} & \rotatebox{90}{moto} & \rotatebox{90}{bike} & \rotatebox{90}{mIoU} \\
\hline\hline
AdaStruct~\cite{RN133} & 84.3 & 42.7 & 77.5 & 4.7 & 7.0 & 77.9 & 82.5 & 54.3 & 21.0 & 72.3 & 32.2 & 18.9 & 32.3 & 46.7 \\
BDA~\cite{RN166} & 86.0 & 46.7 & 80.3 & 14.1 & 11.6 & 79.2 & 81.3 & 54.1 & 27.9 & 73.7 & 42.2 & 25.7 & 45.3 & 51.4 \\
WSDA~\cite{paul2020domain} & \textbf{92.0} & \textbf{53.5} & 80.9 & 3.8 & 6.0 & 81.6 & \textbf{84.4} & 60.8 & 24.4 & 80.5 & 39.0 & 26.0 & 41.7 & 51.9\\
SIM~\cite{RN135}  & 83.0 & 44.0 & 80.3 & 17.1 & 28.7 & 15.8 & 81.8 & 59.9 & 33.1 & 70.2 & 37.3 & 28.5 & 45.8 & 52.1\\
TPLD~\cite{shin2020two} & 80.9 & 44.3 & 82.2 & 20.5 & 30.1 & 77.2 & 80.9 & 60.6 & 25.5 & 84.8 & 41.1 & 24.7 & 43.7 & 53.5 \\
Seg-U~\cite{RN132} & 87.6 & 41.9 & 83.1 & 31.3 & 19.9 & 81.6 & 80.6 & 63.0 & 21.8 & 86.2 & 40.7 & 23.6 & 53.1 & 54.9 \\
FDA~\cite{RN180}  & 79.3 & 35.0 & 73.2 & 19.9 & 24.0 & 61.7 & 82.6 & 61.4 & 31.1 & 83.9 & 40.8 & \textbf{38.4} & 51.1 & 52.5     \\
ProDA~\cite{zhang2021prototypical} &87.1 & 44.0 & 83.2 & \textbf{45.8} & \textbf{34.2} & \textbf{86.7} & 81.3 & 68.4 & 22.1 & \textbf{87.7} & \textbf{50.0} & 31.4 & 38.6 & 58.5 \\
MFA(ours) & 85.4 & 41.9 & \textbf{84.1} & 22.2 & 23.9 & 83.6 & 80.7 & \textbf{71.5} & \textbf{35.8} & 86.6 & 47.6 & 37.2 & \textbf{62.5} & \textbf{58.7}\\
\hline
ProDA*~\cite{zhang2021prototypical} & 87.8 & 45.7 & 84.6 & 54.6 & 37.0 & 88.1 & 84.4 &  74.2 &  24.3 & 88.2 & 51.1 & 40.5 & 45.6 & 62.0 \\
MFA*(ours)  & 81.8 & 40.2 & 85.3 & 38.0 & 33.9 & 82.3 & 82.0 & 73.7 & 41.1 & 87.8 & 56.6 & 46.3 & 63.8 & \textbf{62.5} \\
\hline
\end{tabular}}
\end{center}
\caption{Results on SYNTHIA-to-Cityscapes. MFA achieves better performance than the other state-of-the-art methods. The symbol ``*'' indicates additional distillation stage is used. }
\label{tab:result_syn}
\vspace{-0.2cm}
\end{table}

\subsection{Ablation Study}

\begin{wraptable}{r}{0.5\linewidth
\centering
\vspace{-10pt}
\scalebox{0.8}{
\begin{tabular}{|l|ccc|cc|}
\hline
 Method & TF & CMF & OOF & mIoU & Gain \\
\hline
 Warm & & & &  45.6  & \\
 ST & & & & 49.4 & 3.8 \\
\hline
TF & \checkmark & & &  50.6 & 5.0 \\
TF$\&$CMF & \checkmark & \checkmark & & 51.7 & 6.1 \\
TF$\&$OOF  & \checkmark & & \checkmark &  52.7 & 7.1 \\
MFA & \checkmark & \checkmark & \checkmark & 55.7 & 10.1\\
\hline
\end{tabular}
}
\vspace{1pt}
\caption{Ablation study on the GTA5-to-Cityscapes adaptation. ST: the basic self-training method without any fusions. TF: temporal fusion by consistency loss. CMF: cross-model fusion by jointly generating offline pseudo labels. OOF: online-offline fusion through online pseudo label supervision.}
\vspace{-6pt}
\label{tab:result_method}
\end{wraptable}

Table~\ref{tab:result_method} investigates the contribution of each component of MFA on GTA5-to-Cityscapes. In the first row, ``Warm'' is the best warm-up model based on domain alignment methods~\cite{RN180,RN135}. ``ST'' is the basic self-training ( without any information fusion). We divide MFA to three key components, \emph{i.e.}, the temporal fusion (TF), the cross-model fusion (CMF) and the online-offline pseudo label fusion (OOF). Accordingly, we draw three observations. First, self-training brings $+3.8\%$ mIoU improvement over the warm-up training. Such improvement is consistent with many other two-stage UDA methods \cite{RN135,RN132,RN180}. Second, all the three components are important for MFA. By incrementally adding the key components, the performance reached 50.6\%, 51.7\% and 55.7\%, respectively. Third, these three fusion strategies adds up to +6.3\% mIoU improvement compared to basic ST. They jointly enable MFA achieve significant superiority against other two-stage UDA methods.

\section{Conclusion}
We propose a self-training method named Multi-Fusion Adaptation (MFA) for domain adaptive semantic segmentation. Through a co-learning framework, MFA integrates three information fusion strategies, \emph{i.e.}, cross-model fusion, temporal fusion, and online-offline pseudo label fusion. These fusions jointly suppress the pseudo label noises and explore informative samples during the self-training procedure. Consequentially, MFA significantly improves adaptive semantic segmentation and sets new state of the art on two popular benchmarks.

\paragraph{Acknowledgments.}
This work was supported by the National Natural Science Foundation of China under Grant 61802380, the National Key Research and Development Program of China under Grant 2019YFB1405100 and the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No. XDA19020500.

\section{Introduction}

This paper considers the unsupervised domain adaptation (UDA) for semantic segmentation. In real-world segmentation tasks, there usually exists a domain gap between the training (source domain) and testing data (target domain), which substantially compromises the segmentation accuracy. Instead of using additional annotated data on the target domain for adaptation, which is notoriously expensive, an alternative way is to adapt the already-learned model through UDA ~\cite{hoffman2018cycada,RN180,RN162}. In another word, we aim to improve the segmentation accuracy on an unlabeled target domain without incurring additional annotation.

A popular pipeline adopted by many state-of-the-art methods~\cite{RN180,RN132,zhang2021prototypical} consists of two training stages, \emph{i.e.}, a warm-up supervised training on the source domain and a sequential self-training on the target domain. Specifically, the first stage trains a warm-up model on the source domain data. For better generalization ability, the warm-up training process is typically assisted with some domain alignment constraints~\cite{RN166,RN135}. Then, the second training stage further adapts the warm-up model to the target domain through self-training~\cite{RN162, zou2019confidence}. The self-training usually uses the warm-up model to assign pseudo labels on the target domain, which are used to re-train (fine-tune) the model.

\begin{figure}
\begin{center}
\includegraphics[width=0.6\textwidth]{images/first_structure.pdf}
\end{center}
\vspace{-20pt}
\caption{The proposed Multiple Fusion Adaptation (MFA) employs a co-learning framework to integrate three information fusions \emph{i.e.}, cross-model fusion, temporal fusion and online-offline pseudo label fusion. The learner (Net A) is co-supervised by the offline pseudo labels, as well as the online labels generated by it co-learner (Net B). To make the online label predictions more stable, MFA smooths the co-learner by temporal average (Mean Net B). Importantly, we design the online pseudo labels to be complementary to the offline pseudo labels, which promotes better fusion effect. In the co-learning framework, Net A and Net B will exchange their role of learner and co-learner. We only present Net A as the learner here for easier understanding.
}
\label{fig: intro}
\end{figure}

This paper proposes a novel and effective Multiple Fusion Adaptation (MFA) method, based on the above-described two-stage UDA pipeline. We employ three basic information fusion to improve the domain adaptation, namely a novel \textbf{online-offline pseudo label fusion}, cross-model fusion and temporal fusion. In MFA, all the three fusion strategies are integrated in a co-learning framework, as illustrated in Figure~\ref{fig: intro}. Each learner is co-supervised by two types of pseudo labels, \emph{i.e.}, the online and the offline pseudo labels. The offline pseudo labels are generated with a popular method ~\cite{RN162}, while the online pseudo labels are generated by the temporal average model of the co-learner. This MFA pipeline has two advantages:

$\bullet$ \emph{The novel online-offline pseudo label fusion.} So far as we know, prior two-stage UDA methods~\cite{RN180,RN132} usually employ the offline pseudo labels. Among the iterations of ``assigning pseudo label'' and ``re-training'', the pseudo labels are updated after several training epochs, yielding the ``offline'' manner. While the offline manner allows additional post-processing and has the advantage of balanced pseudo labels~\cite{RN162}, it is prone to the ignorance of hard and informative samples \cite{RN204}. It is because the offline manner only preserves the most confident predictions among all the target domain data, which are relatively easy. As a remedy, we supplement the offline pseudo labels with online ones, which focus on the relatively hard details (\emph{i.e.}, the informative samples) within each training iteration. Moreover, since the online pseudo labels are generated by the up-to-date model (which has better accuracy than the historical ones), they reduce the exposure to noisy supervision and thus benefit the self-training process.

$\bullet$ \emph{A highly efficient integration of three fusion strategies.} While the cross-model fusion and temporal fusion are quite popular, MFA pays significant efforts to raise the efficiency and effectiveness for integration, and succeeds in injecting all the three strategies into a unified framework. Specifically, MFA employs a co-learning framework consisted of two learners, as illustrated in Figure~\ref{fig: intro}. Each learner (model) in MFA is co-supervised by the offline pseudo labels generated by itself, as well as the online labels generated by its co-learner. To make the online predictions more stable, MFA smooths the co-learner by temporal average. Consequentially, MFA simultaneously enforces information fusion between 1) a model and its co-learner (and vice versa), 2) the up-to-date status and the temporal-averaged status and 3) online and offline pseudo labels. Combining these parallel fusions, MFA suppresses the pseudo label noises in the self-training stage and thus improves the segmentation adaptation.

Equipped with these two advantages, MFA is capable to improve UDA for semantic segmentation. First, the novel online-offline pseudo label fusion enables MFA to retain more informative details for adaptive training. Second, MFA manages a highly efficient integration of the online-offline pseudo label fusion and two commonly-adopted fusions, which further increases the accuracy of the pseudo labels. We evaluate the proposed MFA through extensive experiments. Experimental results show that MFA significantly improves the baseline and achieve performance on par with the state of the art. For example, on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes, MFA achieves $58.2\%$ and $62.5\%$ mIoU, respectively. Moreover, ablation study validates the effectiveness of each component in MFA. The main contributions of this paper are summarized as follows:

\begin{itemize}
\vspace{-2pt}
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
\setlength{\parskip}{0pt}
    \item We propose MFA, an unsupervised semantic segmentation adaptation method based on self-training. MFA efficiently integrates three different information fusion strategies to improve the pseudo-label-based UDA.
    \item Among the three fusion strategies of MFA, the online-offline pseudo label fusion is a novel one specifically designed for adaptive segmentation. The online pseudo labels supplement the self-training with relatively hard and informative samples, which may be easily ignored by the offline pseudo labels.

    \item We conduct extensive experiments to evaluate the proposed MFA. Experimental results show that MFA achieves superior adaptive semantic segmentation and the training cost is relatively low.

\end{itemize}

\section{Related work}

\paragraph{Semantic segmentation adaptation.} We divide the existing UDA semantic segmentation methods into two categories: domain alignment~\cite{hoffman2018cycada,RN180,RN138,RN135} and self-training~\cite{RN162,zou2019confidence,shin2020two,paul2020domain,pan2020unsupervised}, and the existing state-of-art approaches are usually a combination of two methods. The main motivation of domain alignment is to reduce the discrepancy between two domains.
CyCADA~\cite{hoffman2018cycada} uses CycleGan~\cite{zhu2017unpaired} to transfer image style. FDA~\cite{RN180} proposes exchanging the low-frequency component of fourier transform without learning to achieve the same purpose.
In addition, SIM~\cite{RN135} introduces the feature alignment of things and stuff respectively.
In self-training, pseudo label learning~\cite{RN162} is a widely used approach. CBST~\cite{RN162} proposes an iterative pseudo label learning strategy and solve the class imbalance issue by class-independent confidence ranking. The improvement of pseudo label learning is an important research direction. In CRST~\cite{zou2019confidence}, a confidence regularized self-training method is proposed to address the problem of overconfident wrong label.~\cite{shin2020two} presents a two-phase pseudo label densification framework through voting-based and  easy-hard classification based method. In ~\cite{paul2020domain}, weak labels are explored to enhance pseudo label learning. Our work considers suppressing the pseudo label noise through multiple fusion strategies.

\paragraph{Temporal average.} Temporal ensembling~\cite{laine2016temporal} averages the outputs of the network-in-training to increase the prediction accuracy for the unlabelled samples. The mean teacher~\cite{tarvainen2017mean} averages model weights at different training steps to get a teacher model. The teacher model offers supervision signal through consistency constraint on the unlabeled samples. These works show that the temporal average of the deep model is more stable and accurate than the deep model at a single training step.

\paragraph{Learning with noisy labels.}  The information fusion between different models is an effective approach to suppress noises in labels. Co-teaching~\cite{han2018co} cross-trains two networks and let them teach each other given the possibly clean labels by small-loss trick. Co-teaching+~\cite{yu2019does} bridges the "Disagreement" strategy with the Co-teaching to enhance robustness under extremely noisy supervision. In these works, the labels are all available, which is different from the unsupervised domain adaptation problem.

We note that~\cite{RN132,zhang2021prototypical} also consider the noise issue of pseudo labels on semantic segmentation adaptation. ~\cite{RN132} estimates the uncertainty of predictions and reduces the impact of low-confidence samples during pseudo label learning. \cite{zhang2021prototypical} take this issue by exploiting the feature distances from prototypes. Our method tackles the noisy pseudo label problem from a different viewpoint. We use co-learning and integrate multiple fusion strategies to resist the noisy pseudo labels, as well as to retain informative samples. Experimental results show that the proposed MFA marginally surpasses~\cite{RN132, zhang2021prototypical}.

\section{Approach}
The proposed MFA adopts the popular UDA pipeline of two-stage training, \emph{i.e.}, a warm-up training on the source domain and a following self-training on the target domain. We first give a formal description of the two-stage UDA pipeline in Section \ref{sec: pre}.
Based on this pipeline, MFA improves the adaptive segmentation through multiple fusions in the self-training stage, as illustrated in Figure~\ref{image:learn_step}. Basically, MFA uses two independent models (Net A and Net B) to set up a co-learning framework. Both models have the dual role of learner and co-learner. Before we collaboratively fine-tune them through self-training, we combine the two warm-up models to generate offline pseudo labels on the target domain (Section \ref{sec: offline}). During self-training, MFA smooths each model through temporal moving average and gets a corresponding ``mean net'' (\emph{i.e.}, Mean Net A and Mean Net B in Figure~\ref{image:learn_step}).  The function of Mean Net (or the temporal moving average operation) is two-fold. First, temporal moving average stabilizes the update of each mean net , therefore making the online predictions more stable. Second, according to the discovery in semi-supervised learning~\cite{tarvainen2017mean}, temporal moving average benefits from the ensemble of multiple models and thus maintains higher prediction accuracy. Given the current training mini-batch , each mean net predicts a respective set of online pseudo labels (Section \ref{sec: online}). Finally, MFA enforces co-supervision on each model. Specifically, each learner is co-supervised by the offline pseudo labels, as well as the online ones generated by its co-learner (Section \ref{sec: co-supervision}).

\subsection{Preliminaries on Two-stage UDA} \label{sec: pre}
In domain adaptive segmentation, we have two datasets belonging to different domains, \emph{i.e.}, the source domain and the unlabeled target domain. The source domain dataset is denoted as $D_{S}=\left \{ x_{S}^{i}, y_{S}^{i} \right \}_{i=1}^{N_{S}}$, where $x_{S}\in\mathbb{R}^{H \times W \times 3}$ is a color image in source domain, $N_{S}$ is the number of source data and $y_{S} \in \mathbb{R}^{H \times W}$ is the corresponding semantic map. The definition of the target domain dataset $D_{T}=\left \{ x_{T}^{j}, y_{T}^{j} \right \}_{j=1}^{N_{T}}$ is similar, except that $y_{T}$ is unknown. Let $F$ represents a semantic segmentation network, and $\theta$ stands the parameters of $F$. The goal of the UDA problem is to estimate $\theta$ to minimize the prediction error on the unlabeled target domain.

In two-stage UDA, the parameter $\theta$ of the warm up model are first obtained through training on the source domain (\emph{i.e.}, stage 1). Then in the self-training (\emph{i.e.}, stage 2), given the input sample from target domain, the prediction $\hat{y}_{T} = F\left ( x_{T} \mid \theta\right )$ is the predicted class probability map, where $\hat{y}_{T}\in\mathbb{R}^{H \times W \times C}$  and $C$ is the number of classes. And $\mathop{\max}\left(\hat{y}_{T}^{c}\right)\in\mathbb{R}^{H \times W}$ is the prediction confidence map. The one-hot map of pseudo labels is obtained by:%
\begin{equation}
\label{equ:option5}
\hat{P}\left ( x_{T} \mid \theta \right )=\text{one-hot}\left(\mathop{\arg\max}_{c}F\left ( c \mid x_{T}, \theta\right)\right)
\end{equation}%
In the standard self-training strategy, an optional way is to retrain the initialized model multiple times by merging the pseudo label data with the source data, which is applied in~\cite{RN135,RN180}. However, this strategy needs to reinitialize $F$ to start training, which is very time-consuming. Therefore, we choose another way used in~\cite{RN132} as our baseline, \emph{i.e.}, fine-tune the warm-up model on the pseudo labels. The loss function for self-supervision is formulated as follows:%
\begin{equation}
\label{equ:option2}
\mathcal{L}_{self}(x_{T},\theta) =-\mathop{\sum}_{batch}m \cdot \hat{p}_{T} \cdot \log\left(F\left ( x_{T} \mid\theta\right )\right)
\end{equation}%
Where $\hat{p}_{T}$ is obtained by $x_{T}$ and  $\theta$ in Equation~\ref{equ:option5}. And $m \in \mathbb{R}^{H \times W}$ is a binary mask for filtering out the unreliable pseudo labels. Specifically, if $m_{h, w}=1$, then we have $\hat{P}_{T, h, w}$ selected for training. In contrast, if $m_{h, w}=0$, the corresponding pseudo label is regarded as unreliable and thus ignored.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{images/second_structure.pdf}
\caption{\textbf{\textcolor{red}{(a)}} The overall framework of the proposed Multiple Fusion Adaptation (MFA). MFA collaboratively trains two models (Net A and Net B), with two supervision signals (Section \ref{sec: co-supervision}), \emph{i.e.}, the offline and online pseudo labels. The offline pseudo labels are generated by a popular baseline CBST \cite{RN162}, as introduced in Section \ref{sec: offline}. The online pseudo labels for Net A are generated by the temporal average of Net B (mean Net B), and vice versa (Section \ref{sec: online}).

\textbf{\textcolor{red}{(b)}} Some examples of the online and offline pseudo labels.
We highlight some regions with red bounding boxes to draw attention of the complementarity between these labels. For example, in the first column, the offline pseudo labels omit some important details about the pedestrians, and the online pseudo labels make up for this problem. In the second column, the person riding a motorcycle is recalled by online pseudo labels.}
\label{image:learn_step}
\end{figure}
\subsection{Offline Pseudo Label}\label{sec: offline}
In Equation~\ref{equ:option2}, the pseudo labels are typically selected in an offline manner, \emph{i.e.}, they will not be instantly updated during model optimization.
MFA combines two different warm-up models for offline pseudo label prediction $\tilde{y}_{T}$.
Consequentially, the offline pseudo labels benefit from model ensemble. Given the raw predictions, we use the CBST~\cite{RN162} method to select the training samples (pixels) by:%
\begin{equation}
\label{equ:option7}
m =
\begin{cases}
1  & \text{ if }\, c=\mathop{\arg\max}\left ( \tilde{y}_{T}^{c} \right )\, \& \, \mathop{\max}\left(\tilde{y}_{T}^{c}\right)>\tau_{c} \\
0 & otherwise
\end{cases}
\end{equation}%
where $c\in \left [ 0, C\right )$ is the class index, and $\tau_{c}$ is the thresholds for the corresponding class. Following~\cite{RN162}, we set each threshold $\tau_{c}$ to ensure class balance, \emph{i.e.}, all the classes has an identical proportion of the selected pixels.

The offline manner of CBST has the advantage of stabilizing supervision signals and avoiding the gradual dominance of large classes~\cite{RN162}. In MFA, the ensemble of two warm-up models further benefits the accuracy of the pseudo labels. However, the offline manner is prone to the problems of \emph{ignorance of hard samples} \cite{RN204} and \emph{longer exposure to the potential noisy labels}. We thus introduce the online pseudo labels as a supplementary.

\subsection{Online Pseudo Label} \label{sec: online}
\paragraph{Temporal average.} MFA instantly assigns online pseudo labels to images in the current mini-batch. In another word, the online pseudo labels do NOT require a post-processing over the whole target domain samples. To alleviate the potential instability issue, MFA first smooths each model through temporal average to generate mean net, which is formulated as:
\begin{equation}
\label{equ:option3}
\theta_{t}^{mean} = \alpha \theta_{t-1}^{mean}+(1-\alpha)\theta_{t}
\end{equation}%
Where $\alpha$ is a smoothing coefficient hyper-parameter, $\theta$ denotes the model parameters and $t$ is the training step. Given the mean net with parameters $\theta^{mean}$ and an input image $x_T$, MFA generates the corresponding online pseudo labels by Equation~\ref{equ:option5}.

\begin{wrapfigure}{R}{0.55\textwidth}
 \vspace{-9pt}
    \begin{minipage}{0.55\textwidth}
      \begin{algorithm}[H]
        \caption{Online CBST}
        \label{alg:cbst}
        \textbf{Input}: Mean Net $F(\theta_{A})$ and $F(\theta_{B})$, minibatch target data $x_{b}$. \\
        \textbf{Parameter}: Ratio $ \varphi $ of selected pseudo labels.\\
        \textbf{Output}: $LP_{b}^{1}$ and $LP_{b}^{2}$ from $\theta_{A}$ and $\theta_{B}$, respectively.
        \begin{algorithmic}
            \STATE $P_{b}^{1} = F\left ( x_{b} \mid \theta_{A} \right )$
            \STATE $P_{b}^{2} = F\left ( x_{b} \mid \theta_{B} \right )$
            \STATE $LP_{b}^{1} = \arg\max\left ( P_{b}^{1},axis=1\right ) $
            \STATE $LP_{b}^{2} = \arg\max\left ( P_{b}^{2},axis=1\right ) $

            \FOR {i=1\ to\ 2}

            \STATE $MP_{b}^{i} = \max\left ( P_{b}^{i},axis=1\right ) $
            \FOR {c=1\ to\ C}
            \STATE $MP_{c, b}^{i} = MP_{b}^{i}\left [ LP_{b}^{i}==c \right ]$
            \STATE $M_{c}^{i} = sort\left ( MP_{c, b}^{i}, order=descending \right ) $
            \STATE $len_{c}^{i} = length(M_{c}^{i})\times \varphi$
            \STATE $\tau_{c}^{i} = M_{c}^{i}\left [ len_{c}^{i} \right ] $
            \STATE $LP_{b}^{i}\left[ LP_{b}^{i}==c \ \& \ LP_{b}^{i} < \tau_{c}^{i}  \right] = 255$
            \ENDFOR
            \ENDFOR
            \STATE \textbf{return} $LP_{b}^{1}$, $LP_{b}^{2}$
        \end{algorithmic}
      \end{algorithm}
    \end{minipage}
\end{wrapfigure}

\paragraph{Online CBST.} The online pseudo labels require to be filtered as well, so as to remove the labels with relatively low confidence. To this end, we propose a novel Online-CBST.

It is similar to the original CBST, except that it uses the data in current mini-batch (rather than the whole training data) as the reference for label selection. In other words, we set a respective filtering threshold for each class, so that all the classes have equal proportion of pseudo-labeled samples in current mini-batch. In analogy to the original CBST, we design the Online-CBST to have linearly-increasing proportion $\varphi(t) \in \left[\rho_{min}, \rho_{max} \right]$, which is the desired proportion in the $t-$th training step. $\rho_{min}$ and $\rho_{max} $ are the pre-defined minimum and maximum proportions.
Initially, the accuracy of each warm-up model is relatively low. So we use a small $\varphi(0)=\rho_{min}$ to retain the most confident pseudo labels and abandon the others.

Since the online pseudo labels are generated by the up-to-date model, they reduce the exposure to noisy supervision. Moreover, Online CBST is beneficial for recalling the relatively hard details (\emph{i.e.}, the informative samples) within each training iteration. In Section~\ref{sec:Role of Online}, we analyzed this in more detail. Given the online pseudo labels $P(x_T\mid\theta_A^{mean})$ and $P(x_T\mid\theta_B^{mean})$, the Online-CBST correspondingly generates two masks $m_A$ and $m_B$ for selecting the pixels. The pseudo code for generating the online labels is to be accessed in Algorithm~\ref{alg:cbst}.

\subsection{Co-supervision in MFA} \label{sec: co-supervision}

Given both the offline and online pseudo labels, MFA enforces a co-supervision on each learner in Figure~\ref{image:learn_step} (a). We illustrate the loss functions for such co-supervision as follows.

To cooperate with the offline pseudo labels, MFA uses the loss function $\mathcal{L}_{self}$ defined by Equation~\ref{equ:option2} for self-supervision on both Net A and Net B. As for the online pseudo labels, MFA uses $P(x_T\mid\theta_A^{mean})$ (predictions from Net A) for supervising Net B and uses $P(x_T\mid\theta_B^{mean})$ for supervising Net A, yielding the so-called cross-model supervision. The detailed loss functions are formulated as:%
\begin{equation}
\label{equ:option9}
\begin{aligned}
&\mathcal{L}_{cross}(\theta_{A}, \theta_{B}^{mean}) = -\mathop{\sum}_{batch}m_{\text{B}}\cdot P\left ( x_{T} \mid \theta_{B}^{mean} \right )   \cdot \log \left (F\left ( x_{T} \mid\theta_{A}\right )\right ) \\
& \mathcal{L}_{cross}(\theta_{B}, \theta_{A}^{mean}) = -\mathop{\sum}_{batch}m_{\text{A}}\cdot P\left ( x_{T} \mid \theta_{A}^{mean} \right ) \cdot \log \left (F\left ( x_{T} \mid\theta_{B}\right ) \right )
\end{aligned}
\end{equation}%
in which $m_B$ ($m_A$) is the selection-mask for $P(x_T\mid\theta_B^{mean})$ ($P(x_T\mid\theta_A^{mean})$) generated by the proposed Online-CBST.

Besides the co-supervision with online and offline pseudo labels, MFA enforces a respective consistency constraint between each learner and its temporal average. Similar to mean teacher~\cite{tarvainen2017mean}, the consistency loss is the expected distance between the prediction of the model and the prediction of the temporal average model, which is formulated as:%
\begin{equation}
\label{equ:option4}
\mathcal{L}_{cst}(\theta, \theta^{mean}) = \mathbb{E}_{x}\left [ \left \| F\left ( x_{T} \mid\theta^{mean}\right ) - F\left ( x_{T} \mid \theta\right ) \right \| \right ]
\end{equation}%
In summary, MFA sums up all the losses to collaboratively train Net A and Net B by:%
\begin{equation}
\label{equ:option10}
\begin{aligned}
\mathcal{L}_{all} = &\mathcal{L}_{self}\left(x_{T},\theta_{A}\right)+\mathcal{L}_{self}(x_{T},\theta_{B}) \\
 &+ \lambda_{cst} \left(\mathcal{L}_{cst}\left(\theta_{A},\theta_{A}^{mean}\right)+\mathcal{L}_{cst}\left(\theta_{B},\theta_{B}^{mean}\right)\right) \\
 &+  \lambda_{cross}\left(\mathcal{L}_{cross}\left(\theta_{A}, \theta_{B}^{mean}\right)+\mathcal{L}_{cross}\left(\theta_{B}, \theta_{A}^{mean}\right)\right),
\end{aligned}
\end{equation}%
in which $\lambda_{cst}$  and $\lambda_{cross}$  are the weighting factors for $\mathcal{L}_{cst}$  and $\mathcal{L}_{cross}$, respectively.

\section{Experiments}
\subsection{Datasets}
We evaluate the proposed MFA under two widely adopted cross-domain segmentation settings, \emph{i.e.}, GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes.
GTA5 and SYNTHIA are both synthetic datasets. The GTA5~\cite{richter2016playing} dataset consists of 24,966 synthesized images of resolution $1914 \times 1052$. Same as existing works, we evaluate our method on 19 common categories shared by GTA5 and Cityscapes. The SYNTHIA~\cite{ros2016synthia} dataset has 9,400 synthesized images of resolution $1280 \times 720$ with fine annotations. Following~\cite{RN135,RN180}, we report the per-class IoU and mIoU on the 13 common categories shared by SYNTHIA and Cityscapes.

Cityscapes is a real-world semantic segmentation dataset~\cite{cordts2016cityscapes}, which consists of $5,000$ images of resolution $2048 \times 1024$ with pixel-level annotations.
It is split into a training set, validation set and test set with $2,975$, $500$ and $1,525$ images, respectively. In line with the standard evaluation setting, we use the $2,975$ training images (without the ground-truth labels) as target domain images, and then evaluate the domain adaptive segmentation accuracy on the validation set.

\subsection{Implementation Details} \label{section:architecture}
Following \cite{RN135,RN180}, we use DeepLabV2~\cite{chen2017deeplab} based on ResNet101 \cite{he2016deep} as the backbone model. We recall that MFA is based on the two-stage UDA pipeline and requires warm-up training. To promote the divergence between the initial status of two learners, we adopt two state-of-the-art domain alignment methods proposed by FDA~\cite{RN180} and SIM~\cite{RN135}.
These two methods serves as the strong baseline for MFA. That being said, we will show that MFA achieves significant improvement over these (\emph{e.g.}, +10.1\% mIoU on GTA5-to-Cityscapes), which results in the state-of-the-art performance. Moreover, MFA is compatible to any warm-up models and is potential to benefit from the future progress in domain alignment.

We use SGD optimization strategy with momentum 0.9. We initialize the learning rate to $2e^{-4}$, and adjust it according to the poly learning rate scheduler with a power of 0.9. As for the hyper-parameters, Equation~\ref{equ:option3} has $\alpha = 0.99$ for temporal average, and Online-CBST has $\rho_{min}=0.2$, $\rho_{max}=0.7$. Moreover, we set $\lambda_{cst}=1.0$, $\lambda_{cross}=0.5$ for Equation~\ref{equ:option10}.

\begin{table}
\begin{center}
\resizebox{\textwidth}{20mm}{
\begin{tabular}{|l|cccccccccccccccccccc|}
\hline
Method & \rotatebox{90}{road} & \rotatebox{90}{sdwk} & \rotatebox{90}{bldng} & \rotatebox{90}{wall} & \rotatebox{90}{fence} & \rotatebox{90}{pole} & \rotatebox{90}{light} & \rotatebox{90}{sign} & \rotatebox{90}{veg} & \rotatebox{90}{trrn} & \rotatebox{90}{sky} & \rotatebox{90}{psn} & \rotatebox{90}{rider} & \rotatebox{90}{car} & \rotatebox{90}{truck} & \rotatebox{90}{bus} & \rotatebox{90}{train} & \rotatebox{90}{moto} & \rotatebox{90}{bike} & \rotatebox{90}{mIoU} \\
\hline\hline
AdaStruct~\cite{RN133} & 86.5 & 25.9 & 79.8 & 22.1 & 20.0 & 23.6 & 33.1 & 21.8 & 81.8 & 25.9 & 75.9 & 57.3 & 26.2 & 76.3 & 29.8 & 32.1 & 7.2 & 29.5 & 32.5 & 41.4     \\
Cycada~\cite{hoffman2018cycada} & 86.7 & 35.6 & 80.1 & 19.8 & 17.5 & 38.0 & 39.9 & 41.5 & 82.7 & 27.9 & 73.6 & 64.9 & 19.0 & 65.0 & 12.0 & 28.6 & 4.5 & 31.1 & 42.0 & 42.7 \\
WSDA~\cite{paul2020domain} & 91.6 & 47.4 & 84.0 & 30.4 & 28.3 & 31.4 & 37.4 & 35.4 & 83.9 & 38.3 & 83.9 & 61.2 & 28.2 & 83.7 & 28.8 & 41.3 & 8.8 & 24.7 & 46.4 & 48.2 \\
BDA~\cite{RN166} & 91.0 & 44.7 & 84.2 & 34.6 & 27.6 & 30.2 & 36.0 & 36.0 & 85.0 & 43.6 & 83.0 & 58.6 & 31.6 & 83.3 & 35.3 & 49.7 & 3.3 & 28.8 & 35.6 & 48.5 \\
SIM~\cite{RN135} & 90.6 & 44.7 & 84.8 & 34.3 & 28.7 & 31.6 & 35.0 & 37.6 & 84.7 & 43.3 & 85.3 & 57.0 & 31.5 & 83.8 & \textbf{42.6} & 48.5 & 1.9 & 30.4 & 39.0 & 49.2     \\
Seg-U~\cite{RN132} & 90.4 & 31.2 & 85.1 & 36.9 & 25.6 & 37.5 & \textbf{48.8} & \textbf{48.5} & 85.3 & 34.8 & 81.1 & 64.4 & 36.8 & 86.3 & 34.9 & 52.2 & 1.7 & 29.0 & 44.6 & 50.3\\
FDA~\cite{RN180} & 92.5  & 53.3  & 82.4  & 26.5  & 27.6  & 36.4  & 40.6  & 38.9  & 82.3  & 39.8  & 78.0  & 62.6  & 34.4  & 84.9  & 34.1  & 53.1  & 16.9  & 27.7  & 46.4  & 50.5\\
TPLD~\cite{shin2020two} & 94.2 & 60.5 & 82.8 & 36.6 & 16.6 & 39.3 & 29.0 & 25.5 & 85.6 & \textbf{44.9} & 84.4 & 60.6 & 27.4 & 84.1 & 37.0 & 47.0 & \textbf{31.2} & 36.1 & 50.3 & 51.2 \\
ProDA~\cite{zhang2021prototypical} & 91.5 & 52.4 & 82.9 & \textbf{42.0} & \textbf{35.7} & 40.0 & 44.4 & 43.3 & \textbf{87.0} & 43.8 & 79.5 & 66.5 & 31.4 & 86.7 & 41.1 & 52.5 & 0.0 & 45.4 & 53.8 & 53.7 \\
MFA(ours)  & \textbf{94.5} & \textbf{61.1} & \textbf{87.6} & 41.4 & 35.4 & \textbf{41.2} & 47.1 & 45.7 & 86.6 & 36.6 & \textbf{87.0} & \textbf{70.1} & \textbf{38.3} & \textbf{87.2} & 39.5 & \textbf{54.7} & 0.3 & \textbf{45.4} & \textbf{57.7} & \textbf{55.7}     \\
\hline
ProDA*~\cite{zhang2021prototypical} & 87.8 & 56.0 & 79.7 & 46.3 & 44.8 & 45.6 & 53.5 & 53.5 & 88.6 & 45.2 & 82.1 & 70.7 & 39.2 & 88.8 & 45.5 & 59.4 & 1.0 & 48.9 & 56.4 & 57.5 \\
MFA*(ours) & 93.5 & 61.6 & 87.0 & 49.1 & 41.3 & 46.1 & 53.5 & 53.9 & 88.2 & 42.1 & 85.8 & 71.5 & 37.9 & 88.8 & 40.1 & 54.7 & 0.0 & 48.2 & 62.8 & \textbf{58.2} \\

\hline
\end{tabular}}
\end{center}
\caption{ Results on GTA5-to-Cityscapes. MFA surpasses all the competing methods. For a fair comparison, ``*'' indicates additional distillation stage is used, which is proposed in~\cite{zhang2021prototypical}. }
\label{tab:result_gta5}
\vspace{-0.2cm}
\end{table}

\subsection{The Effectiveness of MFA}
Table~\ref{tab:result_gta5} compares MFA against several state-of-the-art UDA methods on the GTA5-to-Cityscapes benchmark, from which we draw two observations.
First, comparing MFA against all the competing two stage methods, we find that MFA surpasses all the competing methods by a large margin.
For example, it achieves $55.7\%$ mIoU, which is higher than the strongest competitor ProDA by $+2.0\%$. We achieve the best scores on most categories (11 in 19). Second, under the fair comparison, MFA* presents $58.2\%$ mIoU, with additional model distillation stage~\cite{zhang2021prototypical}. In line with ~\cite{zhang2021prototypical}, we use the SimCLRv2~\cite{chen2020big} pretrained weights and same distillation strategy.

We also compare MFA with competing methods on the SYNTHIA-to-Cityscapes benchmark in the Table~\ref{tab:result_syn} and draw consistent observations as on GTA5-to-Cityscapes. MFA achieves higher mIoU than prior state-of-the-art methods. We report $58.7\%$ mIoU (after self-training stage) and $62.5\%$ mIoU (after distillation stage) on this benchmark.

\subsection{The Efficiency of MFA}
The basic self-training strategy needs iteration of ``assigning pseudo label'' and ``re-training'', which is adopted by~\cite{RN162,RN180}. Benefit from the online-offline fusion, MFA converges faster without multiple iterations and reinitialization. The training in MFA lasts 65 epochs, and is more efficient than other methods~\cite{RN132,RN180,zhang2021prototypical,RN162}. We note that the warm-up models in FDA~\cite{RN180} and the proposed MFA achieve close performance, but MFA achieves 55.7 mIoU after self-training stage (compared to 50.5 in FDA after two round self-training stage). We thus infer that the superiority of MFA is due to the well-engineered self-training with multiple information fusion.

\subsection{The Benefit of Online Pseudo Labels} \label{sec:Role of Online}
We visualize some examples of online and offline pseudo labels in Figure~\ref{image:learn_step} (b). It is clearly observed that online pseudo labels are complementary to the offline ones and focus on the relatively hard details. In the first column, the offline pseudo labels omit the pedestrians, which look small in the image. In contrast, the online pseudo labels succeed in pointing the existence of these pedestrians. In the second column, the person riding a motorcycle is omitted by the offline pseudo labels and is recalled by online pseudo labels. We owe this to two reasons. First, the mean net is updated in time, which is conducive to producing more high-quality pseudo labels along with self-training. Second, since the proposed online CBST focuses on current batch of samples, the relatively difficult samples are more likely to be recalled in the confidence ranking.

\begin{table}
\begin{center}
\resizebox{\textwidth}{20mm}{
\begin{tabular}{|l|cccccccccccccc|}
\hline
Method & \rotatebox{90}{road} & \rotatebox{90}{sdwk} & \rotatebox{90}{bldng} & \rotatebox{90}{light} & \rotatebox{90}{sign} & \rotatebox{90}{veg} & \rotatebox{90}{sky} & \rotatebox{90}{psn} & \rotatebox{90}{rider} & \rotatebox{90}{car} & \rotatebox{90}{bus} & \rotatebox{90}{moto} & \rotatebox{90}{bike} & \rotatebox{90}{mIoU} \\
\hline\hline
AdaStruct~\cite{RN133} & 84.3 & 42.7 & 77.5 & 4.7 & 7.0 & 77.9 & 82.5 & 54.3 & 21.0 & 72.3 & 32.2 & 18.9 & 32.3 & 46.7 \\
BDA~\cite{RN166} & 86.0 & 46.7 & 80.3 & 14.1 & 11.6 & 79.2 & 81.3 & 54.1 & 27.9 & 73.7 & 42.2 & 25.7 & 45.3 & 51.4 \\
WSDA~\cite{paul2020domain} & \textbf{92.0} & \textbf{53.5} & 80.9 & 3.8 & 6.0 & 81.6 & \textbf{84.4} & 60.8 & 24.4 & 80.5 & 39.0 & 26.0 & 41.7 & 51.9\\
SIM~\cite{RN135}  & 83.0 & 44.0 & 80.3 & 17.1 & 28.7 & 15.8 & 81.8 & 59.9 & 33.1 & 70.2 & 37.3 & 28.5 & 45.8 & 52.1\\
TPLD~\cite{shin2020two} & 80.9 & 44.3 & 82.2 & 20.5 & 30.1 & 77.2 & 80.9 & 60.6 & 25.5 & 84.8 & 41.1 & 24.7 & 43.7 & 53.5 \\
Seg-U~\cite{RN132} & 87.6 & 41.9 & 83.1 & 31.3 & 19.9 & 81.6 & 80.6 & 63.0 & 21.8 & 86.2 & 40.7 & 23.6 & 53.1 & 54.9 \\
FDA~\cite{RN180}  & 79.3 & 35.0 & 73.2 & 19.9 & 24.0 & 61.7 & 82.6 & 61.4 & 31.1 & 83.9 & 40.8 & \textbf{38.4} & 51.1 & 52.5     \\
ProDA~\cite{zhang2021prototypical} &87.1 & 44.0 & 83.2 & \textbf{45.8} & \textbf{34.2} & \textbf{86.7} & 81.3 & 68.4 & 22.1 & \textbf{87.7} & \textbf{50.0} & 31.4 & 38.6 & 58.5 \\
MFA(ours) & 85.4 & 41.9 & \textbf{84.1} & 22.2 & 23.9 & 83.6 & 80.7 & \textbf{71.5} & \textbf{35.8} & 86.6 & 47.6 & 37.2 & \textbf{62.5} & \textbf{58.7}\\
\hline
ProDA*~\cite{zhang2021prototypical} & 87.8 & 45.7 & 84.6 & 54.6 & 37.0 & 88.1 & 84.4 &  74.2 &  24.3 & 88.2 & 51.1 & 40.5 & 45.6 & 62.0 \\
MFA*(ours)  & 81.8 & 40.2 & 85.3 & 38.0 & 33.9 & 82.3 & 82.0 & 73.7 & 41.1 & 87.8 & 56.6 & 46.3 & 63.8 & \textbf{62.5} \\
\hline
\end{tabular}}
\end{center}
\caption{Results on SYNTHIA-to-Cityscapes. MFA achieves better performance than the other state-of-the-art methods. The symbol ``*'' indicates additional distillation stage is used. }
\label{tab:result_syn}
\vspace{-0.2cm}
\end{table}

\subsection{Ablation Study}

\begin{wraptable}{r}{0.5\linewidth
\centering
\vspace{-10pt}
\scalebox{0.8}{
\begin{tabular}{|l|ccc|cc|}
\hline
 Method & TF & CMF & OOF & mIoU & Gain \\
\hline
 Warm & & & &  45.6  & \\
 ST & & & & 49.4 & 3.8 \\
\hline
TF & \checkmark & & &  50.6 & 5.0 \\
TF$\&$CMF & \checkmark & \checkmark & & 51.7 & 6.1 \\
TF$\&$OOF  & \checkmark & & \checkmark &  52.7 & 7.1 \\
MFA & \checkmark & \checkmark & \checkmark & 55.7 & 10.1\\
\hline
\end{tabular}
}
\vspace{1pt}
\caption{Ablation study on the GTA5-to-Cityscapes adaptation. ST: the basic self-training method without any fusions. TF: temporal fusion by consistency loss. CMF: cross-model fusion by jointly generating offline pseudo labels. OOF: online-offline fusion through online pseudo label supervision.}
\vspace{-6pt}
\label{tab:result_method}
\end{wraptable}

Table~\ref{tab:result_method} investigates the contribution of each component of MFA on GTA5-to-Cityscapes. In the first row, ``Warm'' is the best warm-up model based on domain alignment methods~\cite{RN180,RN135}. ``ST'' is the basic self-training ( without any information fusion). We divide MFA to three key components, \emph{i.e.}, the temporal fusion (TF), the cross-model fusion (CMF) and the online-offline pseudo label fusion (OOF). Accordingly, we draw three observations. First, self-training brings $+3.8\%$ mIoU improvement over the warm-up training. Such improvement is consistent with many other two-stage UDA methods \cite{RN135,RN132,RN180}. Second, all the three components are important for MFA. By incrementally adding the key components, the performance reached 50.6\%, 51.7\% and 55.7\%, respectively. Third, these three fusion strategies adds up to +6.3\% mIoU improvement compared to basic ST. They jointly enable MFA achieve significant superiority against other two-stage UDA methods.

\section{Conclusion}
We propose a self-training method named Multi-Fusion Adaptation (MFA) for domain adaptive semantic segmentation. Through a co-learning framework, MFA integrates three information fusion strategies, \emph{i.e.}, cross-model fusion, temporal fusion, and online-offline pseudo label fusion. These fusions jointly suppress the pseudo label noises and explore informative samples during the self-training procedure. Consequentially, MFA significantly improves adaptive semantic segmentation and sets new state of the art on two popular benchmarks.

\paragraph{Acknowledgments.}
This work was supported by the National Natural Science Foundation of China under Grant 61802380, the National Key Research and Development Program of China under Grant 2019YFB1405100 and the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No. XDA19020500.

redpajama_set_name: RedPajamaArXiv


Please create your segment(s) first.
See this article https://www.easy-mails.net/article/how-create-segment on how to do that.
To avoid the SPAM folder you need to stop mailing your autoresponder messages after a while.
you can decide to ONLY send the rest of the messages to people who actually OPEN your messages.
Create a new autoresponder campaign, select your Lst, and then choose the segment. For Instance "openers"
Now only engaging subscribers will get your emails.
You can decide to send only broadcast emails to your whole list.
if they are not opening your email after half a year, make sure you delete them from your account.


text:
National University of Galway
EURASHE > About > Partners, Stakeholders and Interlocutors in the EHEA > National University of Galway
Tags: impacct, Organisation, Project
The National University of Ireland Galway (NUIG) is the largest and oldest University in the West of Ireland, with over 17,000 students and more than 2,400 staff. The university is ranked in the top 2% of universities worldwide for teaching and research quality with a distinguished reputation for excellence in the fields of social science, medicine, nursing and health sciences. It is included in the top200 Most International Universities in the world for international outlook (Times Higher Education (THE) World University Rankings2015-2016). NUIG is one of the top universities nationally for graduate employment with 91% of graduates in work or further study within six months of graduating. Within NUIG the Discipline of Health Promotion is a multidisciplinary academic unit incorporating the Health Promotion Research Centre (HPRC). The HPRC was designated as a WHO Collaborating Centre for Health PromotionResearch in 2009 and received re-designation in October 2013. This designation is a recognition of the international standing of the work of the Centre.
Further contacts:
Share National University of Galway
Print friendly National University of Galway

redpajama_set_name: RedPajamaCommonCrawl


Why you should care if you don't care whether someone is black, white, green, or purple.
You Don't Care if Someone Is Black, White, Green, or Purple? You Should!
No more rhetorical human Skittles, please.
The joke is funny, see, because well-oxygenated violet humans do not exist. Green humans do not exist. Yellow with polka dots on your head humans do not exist. When writers scroll through the rainbow in pursuit of ever more outlandish skin tones to assign to their hypothetical fellow man, they almost always end up equating minorities with aliens.
Confession time: I have partaken in this weirdly hued person trend. Once upon a time, I tried to establish a Hammurabi's Code of poetry reviewing.
If you are a person, you can say your opinions about any person's poetry. Straight white men can say things about straight white women's poetry. Black gay men can say things about Asian transgender poetry. Women who turn into polyamorous donkeys on the stroke of midnight can say things about the poetry of little green fellows whose sexual fetishes lie beyond the scope of this analogy.
Later, I begged college students to read the English canon, despite its racist and sexist overtones.
The "stay in your lane" mentality that seems to undergird so much progressive discourse—only polyamorous green people really "get" the "polyamorous green experience," and therefore only polyamorous greens should read and write about polyamorous greens, say—ignores our common humanity.
Let's leave to the therapists my dubious reliance on polyamory as a diversity signifier. (I'm sorry, polyamorists!) I wanted these passages to express, through hyperbolic humor, the irrelevance of race, gender, and sexual orientation as they pertain to reading literature. And I wanted to hint at the dizzying variety and specificity of the identities we can claim for ourselves by positing even more categories, even more shades. But race, gender, and sexual orientation aren't irrelevant. My words reduced the reality of racial difference to a joke. Asian transgender poets endure actual hardships; were-donkeys do not. Like replying to a reminder that "black lives matter" with the cry that "all lives matter," conjuring imaginary green people erases the facts on the ground for minorities—it allows pseudo-creative frippery to elide the real privilege of being white.
Perhaps the worst part, though, is when these words are placed in the mouths of people of color. In Remember the Titans, Denzel Washington's character begins the difficult process of integrating his football team: "Listen up, I don't care if you're black, green, blue, white, or orange, I want all of my defensive players on this side, all players going out for offense over here." There's something lovely and seductive about the belief that, in 1971 Virginia, black equals green equals blue equals white. But naïve rhetoric aside, a good coach in a Nixon-era American high school would care about the racial background of his players; he'd have to. Especially if he were black.


JOANNE M. PIERCE, Courtesy Brewminate.com | The recent dispute over whether Pope Francis denied the existence of hell in an interview attracted wide attention. This isn't surprising, since the belief in an afterlife, where the virtuous are rewarded with a place in heaven and the wicked are punished in hell, is a core teaching of Christianity.
Beginning in the fourth century B.C., after the Greek King Alexander the Great conquered Judea, elements of Greek culture began to influence Jewish religious thought. By time of the first gospels, between 65 and 85 A.D., Jesus refers to the Jewish belief in the eternal fire of Gehenna. Elsewhere, he mentions evildoers' banishment from the kingdom of God and the "blazing furnace" where the wicked would suffer sorrow and despair and "where there will be weeping and gnashing of teeth." Jesus also mentions the Greek Hades when describing how the forces of evil – "the gates of Hades" – would not prevail against the church.
Today, these images seem to be part of a past that the 21st century has outgrown. However, the official textbook of Catholic Christianity, the "Catechism of the Catholic Church," reaffirms the Catholic belief in the eternal nature of hell. It omits the gory details found in earlier attempts to describe the hellish experience but restates that the chief pain of hell is eternal separation from God.


Just curious if anyone has ever pulled a motor off a boat for a transom repair and left the steering/linkages connected while on a engine hoist? I have never removed one before and just wondering if it's even possible? The motor is an 88 special evinrude. My other question is how hard is it to disconnect everything from the motor? Can I do it without screwing everything up? Thanks.
I did had no choice the steering I din't have enough room to slide it out so I pulled the motor then pulled the cable out. My motor is still hanging from the trusses in my garage. Its at the same height so putting it back on the cmc trim will be easy.
I'm doing a transom rebuild now its a little work but in the long run I will know its done the right way and the boat will last another 20 plus years. Just a few weekend and you will have a new boat too, you have a form to help if you have questions. its worth the effort good luck.
brace and gently pound over with whatever mallet you have on hand.
if you don't have a mallet - use a 6" piece of 2x4 lumber and a regular hammer.
if you use a regular steel hammer on that aluminum, it will dent and look terrible.
if you don't anneal the metal first, it will surely crack - in several places.
have you made the new transom panel yet ??
if so, do all your heating prior to installing the wood panel.
for a tutorial - google up "annealing aluminum" before you start.
Last edited by Johnny on 14 Mar 2017, 11:53, edited 1 time in total.
Thank you Johnny! That's what I was looking for. Yes, I have built the transom board and it's ready to be installed. I'll post more pics soon.


I was lucky this week to be part of a photo shoot with a talented and acclaimed local photographer. Sylvaine Poitau – whose work spans over a decade and has covered the likes of BAFTA – arrived on behalf of NWES, supporters of local business and enterprise in Norwich. I had never been professionally photographed before, and was both excited and unsure that my web-cave could play host to anything photogenic.
As it happens, the whole experience was a joy. Sylvaine arrived in the middle of session between myself and Victoria from Dazed Photography. Being a photographer herself, Victoria was used to the light box and tripods, whereas I quickly fazed out the intermittent clicks and flashes. The three of us spent the best part of an afternoon mulling over websites, photography and the local sights, with Sylvaine snapping away at the opportune moment. The results felt effortless and authentic; whatever a photographer's technical skills, the crucial factor seems to be making the subject feel comfortable and at ease. Sylvaine is no exception, and the results show in her work.
I've no idea where or in what form the photos will end up, but I know NWES are undergoing a bit of a revamp. Fingers crossed they mount this web designer on the front window, grinning at passers by in the Norwich market. In any case, I thank them, Sylvaine and Victoria for making all this possible.


Unfortunately, the system we found had been separated from its PSU at some point the in the past - whether it died, or was simply lost we don't know. As a result, powering the unit up to test was complicated slightly. Two options were available here - the power connector (a 3x2 .1" pin header on the side of the system), and the battery connector (a 3-pin header inside the battery compartment). I know from my original system that it's possible to power the system by feeding 6-7.5V into the battery connector; and helpfully the folks at the ACT Sirius 1 user group have put up the power connector pinout. The latter option having a minor caveat that someone over on the vogons.org forum had looked at their PSU and noted that pin 1 was not Gnd, as indicated in the pinout, and did in fact carry a voltage - something to investigate later!
The battery connector was easiest to deal with, as the three pin header is Batt+, Batt-, and a sense line connected to what I /think/ is a thermister shrink-wrapped into the battery pack. This is something I'd investigated with my Grandpa back in the 90s, and we'd determined the system runs fine with that pin disconnected. I hooked up a bench PSU to the battery input, set it for 6V, flicked the power switch, and... nothing. A faint click from the internal speaker, but no lights, nothing on the screen, no drive activity. The system wasn't drawing much current, just a few mA, so clearly something wasn't right. I opened it up to take a look, and decided to start by disconnecting the hard disk controller, which also removes the power connection to the hard disk. This time, when I flicked the switch, we got life! The system started to POST (failed on the CMOS battery - supercap was obviously discharged), but passed everything else, however it was beeping, and the Low Battery LED was illuminated. Increasing the supply voltage to nearer 7.5V cleared this, until the system started to boot from the floppy drive, causing a low battery beep and red light every time it went to seek. Other than this, the system seemed to work fine - keyboard and display are both good, and the floppy drive clearly works.
At this point, I'm suspecting we might have have a hard disk problem - something causing it to load down the rails and prevent the system powering up. First thing to check - plug in the controller but disconnect the disk. In order to do this, the whole system has to be stripped down as the hard disk cable is soldered directly to the controller board, and the hard disk is mounted underneath the motherboard. After pulling the machine apart and removing both hard disk and floppy drive, but leaving the hard disk controller connected, I tried powering it up again - this time it passed the POST, but as expected didn't detect either drive. Yet more things pointing to a hard disk problem - time to go digging.
The next think I tried was to connect the hard disk back up, but leave the floppy drive disconnected. This time the system came up, but the hard disk made a horrible noise - sounded like it was trying to spin up, but something was having issues. The hard disk in this system is a JVC drive, using a proprietary 26-pin interface - I'll leave a lot of the description of the drive to a later post. There isn't much information about the drive out there, but there is a pinout of the connector, and that shows that the last three pairs of pins on the connector carry 5V and 12V power lines (two pairs for 5V, one for 12V). It didn't take much to hook those pins to an ATX PSU and power it up - lo and behold, the drive spun up! After all that, it looks like a power issue - running the hard disk (like floppy seeks), is loading down the 5V rail and causing the system to crash.
With that discovery, the next thing was to attach the hard disk to the laptop, but supply power from the ATX PSU I had on the side of the bench. After doing that, the laptop see the hard disk, and booted right up into MS-DOS 5.0. So after all that, all the key components of the system work, it's just a matter of identifying what's going on with the power supply.
Next was to look at the one piece I didn't expect to work - the battery pack. I know from my original system that this is a shrink-wrapped cardboard tube filled with five sub-C sized NiCd cells. According to the label on this pack, it's an 1800mAh pack, but putting a meter across it showed it to be totally discharged. Just out of interest, I setup my bench supply for 180mA constant-current, and hooked that up to the battery. After watching it for a while, it was taking the current, the voltage was slowly cominh up, and the pack wasn't getting at all warm. After leaving it overnight, I checked it again - still not warm. Taking it off charge, and it showed a voltage of 7.5V - just what you'd expect from a freshly charged 5-cell NiCd pack. Over the next 48 hours that dropped off slightly, down to 6.5V or so, then held steady. Maybe the pack isn't totally dead after-all! Hooked it up to the machine, flicked the power switch, and... nothing. Then again, the system wouldn't power on with 6V on its battery input from the bench supply either. I'm guessing the DC-DC converter on the motherboard needs a recap, so that'll be the next major job to do.
The JVC 26-pin Hard Disk Interface (Part 1) » « What's that? It's Retrochallenge time again?


text:
Bush Appointees Help Murder Terri Schiavo
GOP Cannot Be Trusted on Life
Boulder, CO -- In response to the refusal by the 11th US Circuit Court of Appeals to grant a rehearing in the Terri Schiavo case, America First Party National Chairman Dan Charles issued the following statement:
"Conservatives and Pro-Life activists have long been trapped in the Republican Party by the fiction that the GOP is Pro-Life. Billions of dollars and hundreds of millions of votes have gone to the Democrat and Republican Parties and their various front groups based solely on that fiction. That fiction was laid bare for the world to see by today's 10-2 decision by the 11th Circuit Court of Appeals not to even hear the appeal by Terri Schiavo's parents. Those who voted GOP on the promise of Pro-Life judges must wake up to the fact that they were snookered.
"Republicans control the 11th Circuit, with GOP Presidents appointing 7 of the 12 judges. Six of those Republican appointees--the one Reagan appointee and all five appointed by the current President or his father--voted so as to continue the cruel and inhumane murder by starvation of Terri Schiavo. Four of the five Democrat appointments voted the same way.
"The two judges who voted to give Terri Schiavo a chance to live are Judge Tjoflat, a Ford appointee, and Judge Wilson, a Clinton appointee.
"Among the many GOP appointees voting with the culture of death is Judge William Pryor, appointed by the current President Bush right after he finished prosecuting former Alabama Supreme Court Chief Justice Roy Moore. Judge Pryor argued for Justice Moore's removal from office for the heinous act of putting up a monument that said in part, 'Thou Shall Not Kill.'
"The Associated Press, Reuters, and other news organizations are all reporting the vote was 10-2 to refuse the Schindler's request. Terri's parents are seeking an expedited hearing of their appeal that their daughter be given food and water while they litigate to save her life from the deadly desires of an unfaithful husband with suspect motives. He only 'remembered' that his wife wanted to die after receiving a large malpractice settlement and finding out that his mistress was pregnant with the first of their two children.
"Time is running out for Terri Schiavo. The GOP controlled Florida Legislature and Congress have both failed her. Legal options are nearly gone. Maybe President Bush should declare Terri Schiavo an 'enemy combatant' under authority of the PATRIOT Act, which would allow him to take her into custody and hold her for years, without court redress.
"Or maybe the GOP should turn its attention to the more than 25,000 babies that have been disemboweled since the state started to starve Terri Schiavo to death last Friday.
"The good people of this nation who care about life need to understand the half-hearted 'Pro-Life' gestures by the GOP for what they are: snow jobs to keep the Pro-Life constituency in line. When it comes to stopping the killing, the GOP can't get the real job done because the GOP is not Pro-Life."


Is it just me, or does Ricky Rubio live the most fun-filled existence on the planet? I feel like every video or tweet I see about the guy seems like he's doing something that sounds amazingly fun.
At first I was about to get all judge-y on what kind of TV show has a random puppet just chilling at the bottom of the screen, but then they broke out the Connect Four via basketball set up, and now I am frantically searching eBay to figure out how to gets mine. This would also make a way more enjoyable Price is Right game than the one where the dude yodels up the mountain.
Desperate times call for desperate measures. So, if you are Ricky Rubio and you have to fetch your brother from the airport while the Western Conference Finals are going on, what do you do?
WHATEVER IT TAKES, that's what.
I can only imagine the emotion of a moment like that, when you are forced to give up the one thing that has been a constant in your life since you were a small child. It's no wonder we see so many athletes get choked up in these environs.
Congrats on a wonderful career Brad, and congrats as well to other NBA veterans who have likely played their last games, including Ben Wallace, who has said he will call it quits, and perhaps others like Grant Hill, Jason Kidd, or Kurt Thomas, who might choose the same when all is said and done. I'd include Juwan Howard as well, but he appears to be shooting for Jamie Moyer territory at this point.
A sad day in the NBA, as Ricky Rubio's sensational rookie season comes to an end thanks to a freak knee injury suffered at the tail end of Friday night's loss to the Lakers. Rubio has been even better than advertised in his rookie season, and his dazzling passing and infectious enthusiasm will be sorely missed.
Kevin Durant (@KDTrey5) — Praying the Ricky Rubio has a speedy recovery!
Dwyane Wade(@DwyaneWade) — Ricky Rubio's injury isn't 1 that any basketball fan wanted 2 C. He's had an amazing rookie season & has brought life back 2 a franchise.
Chris Paul (@CP3) — Prayers out to @RickyRubio9 and hoping for a speedy recovery!
Kevin Love (@kevinlove) — Love my teammate and friend @rickyrubio9. Here's to a quick recovery. We will miss you.
Pau Gasol (@paugasol) — All my support to my friend and mate @rickyrubio9 after knowing the hard news about the results on his knee injury.
DeMar DeRozan (@DeMar_DeRozan) — A speedy recovery to Rubio!
Baron Davis — (@Baron_Davis) Rubio smh. Damn ! Praying for you! Wishing you a speedy recovery. You will be stronger than ever!!!!
Greivis Vasquez (@greivisvasquez) — Que triste la lesion de Ricky Rubio, le deseo una pronta recuperacion.
There aren't many players in the league with the kind of court vision and offensive creativity that Rubio possesses. Here's to a complete and total recovery, and hopefully a full season in 2012-13.
When in Rome, right guys?
With that in mind, we're going to keep track of each of the 42 three-plays to see which teams take advantage and which teams fall apart. Up next, the Minnesota Timberwolves, who played three straight from Feb. 28 – Mar. 1.
The Timberwolves take their second crack at the three-fer, having posted a mere 2 points their first go-round Jan 8-10. Could they improve on it this time, with all three games coming on the road?
2 points for the Timberwolves, just like the last time, but they did face a somewhat unique challenge in this one. Not only were all three of their games on the road (which we've seen only a handful of times this season), but each game was their opponent's first game after the All-Star break. Maybe not such a big deal against the Clippers, but certainly a disadvantage against the Suns.
Up next: The defending champion Dallas Mavericks get their lone crack at the threeplay, with three straight contests Mar. 8-10.
Where In The World Is Nikola Pekovic's Deadpan Delivery?
Luke Ridnour, How Do You Rate On The Horry Scale?
Only three game-winning buzzer beaters so far this season. A product of the condensed schedule? Complete coincidence? Whatever it is, here's hoping the second half of the season brings a few more of them.
This shot was pretty easy, thanks in large part by the Jazz deciding that defense wasn't really necessary on their part. After making quick work of Gordon Hayward, Ridnour gets into the paint where Al Jefferson lurks. Jefferson stays at home though, and Luke gets a pretty uncontested look at a floater for the win. To be fair, that shot is pretty delicate regardless of whether it's contested or not, but any point guard worth his salt should have that in their arsenal.
The Timberwolves trailed by 16 with 9:36 left to play, but went on a tear to take a two-point lead with 22.3 seconds left to play. Jefferson then tied the game against his former team on a jumper with 7.0 seconds left to play, and the Timberwolves called timeout and took the ball out at half court. A pretty decent set up for Minnesota, with plenty of time to get a shot off and no penalty for a miss.
Utah and Minnesota are both on the fringe of the playoff chase, and in the loaded West every win counts. This is particularly nice for the Timberwolves, especially given how they lost the other night.
Ridnour gives the traditional two fingers pointed skyward, and the team rallies around to congratulate him by the bench. It's always great when a shorter player does something — always more exciting when one of his teammates picks him up to celebrate.
2 Horrys. All in all a fairly pedestrian buzzer-beater, largely due to the Jazz' defensive indifference, but I'll give it an extra Horry due to both teams' proximity to each other in the race for the 8 spot in the West, plus the terrific fourth-quarter comeback to get them in position to win in the first place.