text:
\section{Conclusion}\label{sec:conclusion}
In this paper, we have introduced an online incremental learning problem for semantic segmentation, which is more practical for real-world applications. Our problem setting assume that a limited memory is available for model learning while the data comes in a streaming manner and the model updates its parameters only once for each incoming batch.
To address this challenging problem, we develop a unified EM learning framework that integrates a re-labeling strategy for missing pixel annotations and an efficient rehearsal-based incremental learning step with dynamical sampling.
Moreover, we also introduce cosine normalization and class-balanced memory to solve the class-imbalanced problem
Extensive experimental results on the PASCAL VOC 2012 and the ADE20K dataset demonstrate the advantages of our methods

\section{Our approach}\label{sec:approach}

In this section, we introduce our learning strategy for incremental semantic segmentation, which aims to continuously learn novel visual concepts for pixel-wise semantic labeling of images.
In particular, given a sequence of semantic classes, we consider the task of building a segmentation model in multiple incremental \remove{steps}\myadd{tasks}, and in each \remove{step}\myadd{task}, the model needs to be updated for an
expanded semantic label space with only limited data memory of previous \remove{steps}\myadd{tasks}.
In this limited-memory setting, our goal is to achieve efficient adaptation of the segmentation model without catastrophic forgetting of previously learned semantic classes.

To this end, we develop an incremental learning approach for deep segmentation networks based on the Expectation-Maximization (EM) framework. Our EM learning method incorporates a re-labeling strategy for augmenting annotations and an efficient rehearsal-based model adaptation with dynamic data sampling within a single framework.
Such an integrated framework enables us to cope with the challenges of catastrophic forgetting, efficient model adaptation, and background concept drift simultaneously and in a principled manner.
Below we start with an introduction of our problem setting in Sec. \ref{subsec:setup}, followed by a formal description of our model architecture in Sec. \ref{subsec:model_arch}.
Then we present our EM-based incremental learning approach in Sec. \ref{subsec:em_framework}.
\subsection{Problem Setup}\label{subsec:setup}
The task of online incremental semantic segmentation typically starts from a base task $\mathcal{T}_0=\{\mathcal{D}^{tr}_0,\mathcal{Y}_0\}$ in which $\mathcal{D}^{tr}_0$ is the training dataset, and $\mathcal{Y}_0$ are the base semantic classes. We denote the {background class} of $\mathcal{T}_0$ as $U_0$, which includes all the irrelevant classes not in $\mathcal{Y}_0$, and $\tilde{\mathcal{Y}}_0=\mathcal{Y}_0\cup \{U_0\}$ are the categories including background class at the initial \remove{step}\myadd{task}.
The initial training set $\mathcal{D}^{tr}_0=\{(\mathbf{x}^0_i,\mathbf{y}^0_i)\}_{i=1}^{|\mathcal{D}^{tr}_0|}$, where $\mathbf{x}_i^0\in\mathbb{R}^{H\times W \times 3}$ and $\mathbf{y}_i^0\in \tilde{\mathcal{Y}}_0^{H\times W}$ are the $i$-th image and its fully-labeled annotation, respectively. For the base task, we assume a segmentation model $\mathcal{S}_0$ has been trained on $\mathcal{D}^{tr}_0$.

During the incremental learning stage, the \remove{system}\myadd{pretrained segmentation model} receives a stream of new semantic class groups $\{\mathcal{C}_t\}_{t\geq 1}$ and their corresponding training data $\{\mathcal{D}^{tr}_t\}_{t\geq 1}$.
For task $t$, the data comes as a stream of mini-batch $B^{tr}_i=(\bm{X}_i, \bm{Y}_i)$  in an online manner where $\bm{X}_i$ means a set of images and $\bm{Y}_i$ is its corresponding annotation.
In other words, the model is only allowed to use the incoming batch $B^{tr}_i$ to update its parameters only once(up to k times).
\myadd{
We assume a small set of exemplars from previous steps (up to $t$) can be stored in an external memory, denoted as $\mathcal{M}_t$, and can be used for training the segmentation model $\mathcal{S}_t$.
Typically, samples are drawn from memory and concatenated with the incoming batch.}
At each incremental task $t$, we assume that only pixels from the novel classes  are labeled, which may be caused by lacking previous labeling expertise and/or limited annotation budget.
Specifically, the incoming training data $B^{tr}_k$ has a form of $\{(\mathbf{x}^t_i,\mathbf{y}^t_i, \mathcal{R}^t_i)\}_{i=1}^{|B^{tr}_k|}$ at task $t$, in which $\mathbf{y}^t_i$ is the partial annotation, and $\mathcal{R}^t_i$ indicates the labeled region in the image $\mathbf{x}^t_i$.
For each pixel $j\in \mathcal{R}^t_i$, we have its label $\mathbf{y}^t_{i,j}\in \mathcal{C}_t$ and otherwise $\mathbf{y}^t_{i,j}$ is unlabeled.

Our goal is to learn an updated segmentation model $\mathcal{S}_t$ to incorporate the novel semantic classes in $\mathcal{C}_t$, and to achieve strong performance on a test data $\mathcal{D}^{ts}_t$.
We note that $\mathcal{D}^{ts}_t$ includes all the semantic classes up to the $t$-th task, denoted as $\mathcal{Y}_t=\mathcal{Y}_0\cup\left(\cup_{\tau=1}^t\mathcal{C}_\tau\right)$, and the corresponding \textit{background class} $U_t$

\begin{figure*}[t]
	\begin{minipage}{0.47\linewidth}
		\centering

		\begin{algorithm}[H]
			\caption{The Method Overview.}
			\hspace*{-0.2em} {\bf Initialize:} Model $\mathcal{S}_0$; Memory Size $|\mathcal{M}|$; Memory $\mathcal{M}_0=$$\text{Random Sample}(\mathcal{D}^{tr}_0;|\mathcal{M}|)$
			\begin{adjustwidth}{-1em}{}
				\begin{algorithmic}
					\For{\texttt{$t=1,2,3...$}}
					\State \textcolor{gray}{// warm start}
					\State Select $D_c$ from $\tilde{\mathcal{D}^{tr}_t} = \{\mathcal{M}_{t-1}, \mathcal{D}^{tr}_t \}$
					\State Extract Conv features $\mathcal{V}_c\!=\!\{\mathcal{F}_{t-1}(\bm{x}_i)\!:\!\bm{x}_i\in D_c \}$
					\State Finetune segmentation head $\mathcal{H}_t$ with $\mathcal{V}_c$
					\State \textcolor{gray}{// incremental learning with EM}
					\While{The budget $\mathcal{B}_t$ is not used up}
					\State Run EM iteration in \textbf{Algorithm} \ref{alg:em}.
					\EndWhile
					\State $\mathcal{M}_t$$=$$\text{Random Sample}(\mathcal{M}_{t-1},\mathcal{D}^{tr}_t;|\mathcal{M}|)$
					\EndFor
				\end{algorithmic}
			\end{adjustwidth}
			\label{alg:overview}
		\end{algorithm}
	\end{minipage}
	\hspace{5mm}
	\begin{minipage}{0.47\linewidth}
		\centering

		\begin{algorithm}[H]
			\caption{The EM Iteration.}
			\hspace*{0in} {\bf Input:} The model $\mathcal{S}$; Step $t$; Batch-size $B$
			\begin{adjustwidth}{-0.9em}{}
				\begin{algorithmic}
					\State \textcolor{gray}{// sample images to build mini-batch $\mathcal{N}_t$}
					\State $\mathcal{N}_t = \emptyset$
					\For{\texttt{$k=1,2,3...,B$}}
					\State Sample the class $c_{k}$ according to $P_s(c)$ (Eq. \ref{eq:3})
					\thickmuskip=0.5\thickmuskip
					\State Denote $D_s\!=\!\{(\bm{x}_l,\bm{y}_l,\mathcal{R}_l)\in \mathcal{M}_{t}|\,\exists j\in \mathcal{R}_l,\,\bm{y}_{l,j}=c_k\}$
					\State Uniformly sample $(\bm{x}_i,\bm{y}_i,\mathcal{R}_i)$ from $D_s$
					\State $\mathcal{N}_t = \mathcal{N}_t \cup \{i\}$
					\State Refine label $\bm{y}_i$ by relabeling (Eq. \ref{eq:4})

					\EndFor
					\State Update model $\mathcal{S}$ with the loss $\mathcal{L}$ (Eq. \ref{eq:5})
					\State Update class sampling probability $P_s(c)$ (Eq. \ref{eq:3})
				\end{algorithmic}
			\end{adjustwidth}
			\label{alg:em}
		\end{algorithm}
	\end{minipage}
\end{figure*}

\subsection{Model Architecture}\label{subsec:model_arch}
<<<<<<< HEAD
For each task $t$, we assume a typical semantic segmentation network architecture, which consists of an encoder-decoder backbone network $\mathcal{F}_t$ extracting a dense feature map and a convolution head $\mathcal{H}_t$ that produces the segmentation score map. Note that our method is agnostic to specific network designs.
=======
For each \remove{step}\myadd{task} $t$, we assume a typical semantic segmentation network architecture, which consists of an encoder-decoder backbone network $\mathcal{F}_t$ extracting a dense feature map and a convolution head $\mathcal{H}_t$ that produces the segmentation score map. Note that our method is agnostic to specific network designs.
>>>>>>> bed53a885c955f5dcdaeca5fdd4a6bf0a87a2bbd

Concretely, given the image $\mathbf{x}_i$, the segmentation network first computes the convolutional feature map $\mathbf{F}_i\in\mathbb{R}^{H\times W\times D}$ using the backbone network, where $D$ is the number of channels. $\mathbf{F}_i$ is then fed into the $1\times 1$ convolution head $\mathcal{H}_t$, which has a form of linear classifier at each pixel location. Denote the parameters of the backbone network as $\mathbf{W}_t^b$, and the parameters of head as $\mathbf{W}^h_t\in \mathbb{R}^{D\times(|\mathcal{Y}_t|+1)}$, our network generates its output $\mathbf{\hat{y}}_i$ as follows,
\begin{align}
&\mathbf{F}_i=\mathcal{F}_t(\mathbf{x}_i;\mathbf{W}^b_t), \\
&p(\mathbf{\hat{y}}_i|\mathbf{x}_i)=\text{\small{Softmax}}(\mathcal{H}_t(\mathbf{F}_i))=\prod_{j\in\Omega_i}p(\mathbf{\hat{y}}_{ij}|\mathbf{F}_{ij};\mathbf{W}^h_t)
\end{align}
where $\Omega_i$ represents the 2D image plane and $\mathbf{F}_{ij}\in \mathbb{R}^{D}$ for the pixel $j$.

\subsection{Incremental Learning with EM}\label{subsec:em_framework}

We now develop an EM learning framework to efficiently train the model $\mathcal{S}_t$ at the \remove{step}\myadd{task} $t$ with a limited memory $\mathcal{M}_t$ and a computation budget $\mathcal{B}_t$. To tackle the stability-plasticity dilemma, our strategy integrates the ideas in rehearsal-based learning and pixel relabeling, which enables us to efficiently re-use the training data for fast model adaptation and cope with background drift caused by the partial annotation.

Specifically, we formulate the problem as learning with latent variables and develop a stochastic EM algorithm to iteratively update the model parameters.
In each iteration, our EM algorithm first samples a mini-batch of training images, then fills in the missing labels and finally trains the model with one step of SGD.
 Below we assume an initial parameter $\bm{\theta}_t$ and describe the details of the three stages in each EM iteration. A complete overview of our algorithm is shown in Algorithm~\ref{alg:overview}.

\paragraph{Dynamic sampling}

The first stage is to sample a mini-batch $\mathcal{N}_t$ of training data for a single step in the subsequent EM update. Our goal is to achieve efficient model updates to learn the new classes but also to retain old ones. To this end, we develop a dynamic sampling strategy that only selects a small set of informative training data for an EM iteration.

Specifically, we devise a category-level sampling strategy that attempts to balance the training samples from the novel classes to be learned and the old ones prone to model forgetting. Concretely, our method maintains a  \textit{class sampling probability} $P_s(c)$ for $\forall c\in \tilde{\mathcal{Y}}_t$, which will be described below. In order to sample a training image, we first sample a class label $c_i$ according to $P_s$, and then uniformly sample an image $\mathbf{x}_i$ that has pixels labeled with $c_i$.
Such a sampling process allows the model to visit certain classes more frequently to improve their learning.

Our key design is to dynamically update the class probability $P_s$ according to the prediction confidence in the outputs of current model $\mathcal{S}_t$.
To achieve this, we first compute an estimate of class confidences, denoted as $\{E(c),c\in\tilde{\mathcal{Y}}_t\}$, at each iteration of EM. Concretely, we use a moving average to collect the statistics from mini-batches as follows,
\begin{align}
E(c) &= \mu E_{{old}}(c) + (1-\mu)E_{mb}(c), \quad  \\
E_{mb}(c) &= \underset{{i\in \mathcal{N}_t, \mathbf{{y}}_{i,j}=c}}{\text{Mean}}\{p(\mathbf{\hat{y}}_{i,j}=c|\mathbf{x}_i;\bm{\theta}_t^{old})\}
\end{align}
where $E_{old}$ and $E_{mb}$ denotes the confidence estimate from last iteration and current mini-batch, respectively, $\mu$ is the momentum coefficient, and $\bm{\theta}_t^{old}$ is the model parameters from last iteration. At the beginning of each incremental step $t$, we initialize $E(c)$ as $0$ for all the class in $\tilde{\mathcal{Y}}$.

Given the class confidences, we define the sampling probability $P_s$ with a Gibbs distribution, which assigns higher sampling probability to the classes with lower confidence:
\begin{align}
P_s(c) = \frac{e^{-\eta E(c)}}{\sum_{k=1}^{|\tilde{\mathcal{Y}}_t|} e^{-\eta E(k)}} \quad \forall c\in \tilde{\mathcal{Y}}\label{eq:3}
\end{align}
where $\eta$ is the hyperparameter to control the peakiness of the distribution.

\paragraph{E-step}

In E-step, we tackle the problem of background concept drift due to the expansion of semantic label space $\mathcal{Y}_t$, and exploit the unlabeled regions in the training data of incremental steps for model adaption. To this end, we treat the labels of the background pixels in $\{\mathcal{D}^{tr}_0\}$ and the unlabeled pixels in $\{\mathcal{D}^{tr}_t\}_{t\geq 1}$ as hidden variables, and use the current model to infer their posterior distribution given the images and semantic annotations. We then use the posterior to fill in the missing annotations for the subsequent model learning in the M-step. By augmenting the training data with those pseudo labels, we aim to further improve the training efficiency and alleviate the problem of inconsistent background annotation.

Specifically, given a training pair $(\mathbf{x}^\tau_i, \mathbf{y}^\tau_i), i\in \mathcal{N}_t$, in our sampled mini-batch, we first compute an approximate posterior of the latent part of $\mathbf{y}^\tau_i$. Here $\tau\in\{0,\cdots,t\}$ indicates the incremental step it comes from. Denote the region with latent labels as $\bar{\mathcal{R}}^\tau_i$, the posterior of its $j$-th pixel can be estimated as
$p(\mathbf{\hat{y}}_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{old})$ based on the current network. We then generate a pseudo labeling for those pixels by choosing the most likely label predictions as follows,
\begin{align}
\mathbf{{y}}^\tau_{i,j} = \underset{\mathbf{\hat{y}}_{i,j}\in\mathcal{\tilde{Y}}_t - \mathcal{C}_\tau}{\arg\max} p(\mathbf{\hat{y}}_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{old}), \quad j\in \bar{\mathcal{R}}^\tau_i
\label{eq:4}
\end{align}
where $\mathcal{C}_\tau$ is the class group from step $\tau$. To filter out unreliable estimation, we only keep the pseudo labels of those pixels with confidence higher than a threshold $\delta$, i.e., $p(\mathbf{{y}}^\tau_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{old})>\delta$. We adopt this hard EM to update our model in the next stage.

\paragraph{M-step}

Our M-step employs the mini-batch training data with augmented annotations to update the model parameters. In order to fully utilize the image annotations, we propose a composite loss consisting of two loss terms as follows, \begin{align}
\mathcal{L} = &- \sum_{i\in \mathcal{N}_t} \sum_{\mathbf{{y}}^\tau_{i,j}\in\mathcal{\tilde{Y}}_t} \log p(\mathbf{{y}}^\tau_{i,j}|\mathbf{x}_i^\tau;\bm{\theta}_t) \\
	&- \gamma\sum_{i\in \mathcal{N}_t} \sum_{j\in\bar{\mathcal{R}}^\tau_i} \log \sum_{\mathbf{{y}}^\tau_{i,j}\notin \mathcal{C}_\tau} p(\mathbf{{y}}^\tau_{i,j}|\mathbf{x}_i^\tau;\bm{\theta}_t)
\label{eq:5}
\end{align}
where $\gamma$ is a weighting coefficient. The first term is the standard cross-entropy loss on every pixel with real annotation or pseudo labels, while the second term penalizes the model on classifying the unlabeled pixels at step $\tau$ into label space $\mathcal{C}_\tau$, which encourages the labeling consistent with the partial groundtruth annotations.

\subsection{Class Balancing Strategies}
We introduce our strategies to deal with the class-imbalance issues in the online class-incremental semantic segmentation problem. We first present the cosine normalization and then introduce the class-balanced exemplar selection strategy in this subsection.

\paragraph{Cosine Normalization}For the prediction of each pixel at $j$, we have a probability vector $p(\mathbf{\hat{y}}_{ij}|\mathbf{F}_{ij};\mathbf{W}^h_t)$ with dimension $|\mathcal{Y}_t|+1$ where k-th element is as follows
\begin{align}
	p(\mathbf{\hat{y}}^k_{ij}|\mathbf{F}_{ij};\mathbf{W}^h_t) = \frac{e^{\tau d(\bm{u}_k , \mathbf{F}_{ij})}}{\sum_{c=1}^{|\mathcal{Y}_t|+1} e^{\tau d(\bm{u}_c , \mathbf{F}_{ij})}}
\end{align}
where $\tau$ is the temperature used to control the sharpness of the softmax distribution, $u_k\in\mathbb{R}^D$ is the k-th column of classifier weight $\mathbf{W}^h_t$, and $d$ means the cosine distance metric.
It is worth noting that we are the first to adopt cosine normalization in the online class incremental learning problem, even though there has been works adopting it in many other problems\cite{luo2018cosine,qi2018low}.

\paragraph{Class-balanced Exemplar Selection}

We extend the CBRS(class-balanced reservoir sampling)~\cite{} into the incremental semantic segmentation problem by considering the multiple instances of different classes in the image.
Therefore, we update the memory to make it as class-balanced as possible.
We pre-allcoate a size for each class.We allocate nearly same memory size for each class.
The streaming data is highly temporally correlated.
We apply reservoir sampling for each class.

\section{Introduction}
\begin{figure}[t]
	\centering

	\resizebox{0.48\textwidth}{!}{
	\includegraphics[width=\textwidth]{imgs/ad.pdf}
   }
\caption{Overview of our online incremental semantic segmentation setting. Each incremental task adds images with annotations of new classes only for model adaptation, and a small set of data is saved for subsequent learning. We adopt the single-pass setting in which the data come as a stream of batches and can only be used once to update the model parameters. After each task, the model aims to segment all the semantic classes seen so far. }	\label{fig:intro}
\end{figure}
Semantic segmentation of visual scenes has recently witnessed tremendous progress  thanks to pixel-level representation learning based on deep convolutional networks. Most existing works, however, assume a close-world setting, in which all the semantic classes of interest are given when a deep segmentation network is learned. This assumption, while simplifying the task formulation, can be restrictive in real-world applications, such as medical image analysis, robot sensing, and personalized mobile Apps~\cite{li2019efficient}, where novel visual concepts need to be incorporated later on or continuously in deployment. To address this, a promising strategy is to incrementally learn semantic segmentation models~\cite{michieli2019incremental,cermelli2020modeling}.

In this work, we focus on the problem of incrementally learning semantic segmentation model from \textit{a data stream}~\cite{aljundi2019mir,aljundi2019gss}. In contrast to the offline learning, our online learning task is bounded by the run-time and only allows single-pass through the data.
Here we assume that, to reduce expensive labeling cost for semantic segmentation, we only receive annotations for the pixels of novel classes at their arrival as in \cite{cermelli2020modeling}. During training, we also assume that the system is able to
save a subset of previously-seen training data as memory for rehearsal. This enables us to adapt the learned segmentation model to new label spaces with low annotation and training cost, and to cope with the scenarios that only limited legacy data resource can be stored due to privacy issue or data memory limitation.
Fig.~\ref{fig:intro} illustrates the typical problem setting of this incremental semantic segmentation task.
We note that most prior attempts on the incremental learning for semantic segmentation either adopt the offline learning setting without considering memory, or only focus on domain-specific tasks~\cite{ozdemir2018learn,tasar2019incremental,cermelli2020modeling}. By contrast,
we aim to address the online incremental semantic segmentation with limited memory, which is more practical for many real-world scenarios.

There are several challenges in this online class incremental semantic segmentation problem given limited memory resource.
First, continuous learning of novel visual concepts typically involves the so-called \textit{stability-plasticity dilemma}, in which a model needs to quickly learn new concepts and meanwhile overcome catastrophic forgetting on learned visual knowledge~\cite{rebuffi2017icarl}.
In addition, the segmentation models have to cope with \textit{background concept drift} as the background class changes due to the introduction of novel semantic classes, which further increases the difficulty of fast model adaption.
Moreover, there also exists severe class-imbalanced problem due to varying co-occurrence of semantic classes in the data stream and small-sized memory.

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{imgs/ourmethod.pdf}
\caption{Overview of our framework. Each incremental step begins with a new training batch $\mathcal{N}_{t}^{k}$.
	Our method first samples a batch with the same size as the new batch from a class-balancing memory with a class probability $P_{s}$,
	which is dynamically updated at every step according to the prediction confidence of each class.
	The sampled batch is combined with the incoming data to build the training batch.
	We then update our model based on the EM learning framework: We fill in the missing annotations using confident predictions of the trained model, which produces the pseudo-masks for the training batch at E-step. At M-step,
	we train the model by optimizing the composite loss between the model predictions and the pseudo masks with cosine normalization.
	At the end of this incremental step, we use class-balanced exemplar selection to update the memory.
}
\label{fig:inc_algorithm}
\end{figure*}
Our goal is to tackle all three aforementioned challenges in the online incremental learning of segmentation, aiming to achieve high learning efficiency with robustness towards catastrophic forgetting and concept drift.
To this end, we propose a novel rehearsal-based deep learning strategy for building a segmentation network in an incremental manner.
Our key idea is to train a deep network by utilizing incoming data batches at each incremental step with a \textit{dynamic sampling} policy, which concentrates on informative samples from previous steps.
This allows us to alleviate the catastrophic forgetting problem with low training cost.
In addition, to improve the data efficiency and overcome the background drift, we introduce a re-labeling strategy in each incremental step, which fills in the missing annotations and updates the background class using confident predictions of the trained model.
Furthermore, we integrate two class-balancing strategies, the cosine normalization and class-balanced exemplar selection, into the network training in each step to cope with unbalanced data.

Formally, we develop a unified learning approach that integrates the above strategies into an Expectation-Maximization (EM) based learning framework. Our method starts with a deep network trained on a set of base categories, and sequentially learns novel visual classes over incremental steps. In each step, our EM learning iterates through three stages: a) sampling a small set of training data, b) filling in missing labels of pixels, and c) updating the segmentation network using the dataset with mixed true and pseudo labels.
After the parameter update, we adopt a class-balanced reservoir sampling strategy to update the exemplar set in the memory, which is then used for the next incremental step.

We evaluate our model on two challenging benchmarks, including PASCAL VOC 2012 and ADE20K datasets. Our empirical results and ablation study show that the proposed model achieves superior performance over prior incremental approaches, demonstrating the efficacy of our method. To summarize, the main contributions of our work are three-fold:

\begin{itemize}[leftmargin=5mm,topsep=0pt,itemsep=0pt]
	\item We introduce a new online incremental learning problem for semantic segmentation, and our approach achieves the state-of-the-art results on two challenging benchmarks.
	\item We develop a unified EM framework that integrates a re-labeling step and a rehearsal-based training with dynamic sampling, promoting fast adaptation to novel classes and better alleviating catastrophic forgetting.
	\item To cope with imbalanced data in the online setting, we introduce cosine normalization and class-balanced reservoir sampling for incremental training of the segmentation networks.

\end{itemize}

\section{Our approach}\label{sec:approach}

In this section, we introduce our online incremental learning strategy for semantic segmentation, which aims to continuously learn novel visual concepts for pixel-wise semantic labeling of images.
In particular, given a sequence of semantic classes, we consider the problem of building a segmentation model in multiple incremental tasks, each of which expands semantic label space with new arriving classes and needs to update the model with only limited data memory of previous tasks.
In this limited-memory setting, our goal is to achieve efficient adaptation of the segmentation model without catastrophic forgetting of previously learned semantic classes.

To this end, we develop an Expectation-Maximization (EM) framework for incrementally learning a deep segmentation network. Our EM learning method incorporates a re-labeling strategy for augmenting annotations and an efficient rehearsal-based model adaptation with dynamic data sampling within a single framework.
In addition, we introduce a robust training procedure for adapting the segmentation model based on cosine normalization and class-balanced memory.
Such an integrated framework enables us to cope with the challenges of stability-plasticity dilemma, background concept drift and class-imbalance simultaneously in a principled manner.
Below we start with an introduction of our problem setting in Sec.~\ref{subsec:setup}, followed by a formal description of our model architecture in Sec.~\ref{subsec:model_arch}.
Then we present our EM-based incremental learning approach in Sec.~\ref{subsec:em_framework}.
Finally, we show our class-balancing strategies in Sec.~\ref{subsec:class_balanced}.
\subsection{Problem Setup}\label{subsec:setup}
The online incremental learning of semantic segmentation typically starts from a segmentation model $\mathcal{S}_0$, which is trained on a base task $\mathcal{T}_0=\{\mathcal{D}^{tr}_0,\mathcal{Y}_0\}$ where $\mathcal{D}^{tr}_0$ is the training dataset, and $\mathcal{Y}_0$ are the base semantic classes. We denote the {background class} of $\mathcal{T}_0$ as $U_0$, which includes all the irrelevant classes not in $\mathcal{Y}_0$, and $\tilde{\mathcal{Y}}_0=\mathcal{Y}_0\cup \{U_0\}$ are the categories including background class at the initial task.
The initial training set $\mathcal{D}^{tr}_0=\{(\mathbf{x}^0_i,\mathbf{y}^0_i)\}_{i=1}^{|\mathcal{D}^{tr}_0|}$, where $\mathbf{x}_i^0\in\mathbb{R}^{H\times W \times 3}$ and $\mathbf{y}_i^0\in \tilde{\mathcal{Y}}_0^{H\times W}$ are the $i$-th image and its fully-labeled annotation, respectively.

During the incremental learning stage, the initial segmentation model receives a stream of new semantic class groups $\{\mathcal{C}_t\}_{t\geq 1}$ and their corresponding training data $\{\mathcal{D}^{tr}_t\}_{t\geq 1}$ where $t$ indicates incremental steps.
In each incremental step $t$, the training data also comes as a stream of mini-batches $\mathcal{N}^{k}_t \subset \mathcal{D}^{tr}_t$ where $k=1,\cdots,K$ and K is the number of min-batches. As in~\cite{aljundi2019mir, chaudhry2018efficient}, our online learning setting only allows the model to perform one-step parameter update using the incoming batch $\mathcal{N}^{k}_t$.
Here we assume that only pixels from the novel classes are labeled, which may be caused by lacking previous labeling expertise and/or limited annotation budget.
Specifically, the incoming training data $\mathcal{N}^{k}_t$ has a form of $\{(\mathbf{x}^t_i,\mathbf{y}^t_i, \mathcal{R}_i^t)\}_{i=1}^{|\mathcal{N}^{k}_t|}$ at step $t$, in which $\mathbf{y}^t_i$ is the partial annotation, and $\mathcal{R}^t_i$ is derived from $\mathbf{y}^t_i$ and indicates the labeled region in the image $\mathbf{x}^t_i$.
For each pixel $j\in \mathcal{R}^t_i$, we have its label $\mathbf{y}^t_{i,j}\in \mathcal{C}_t$ and otherwise $\mathbf{y}^t_{i,j}$ is unlabeled.

Moreover, we assume a small set of exemplars $\mathcal{M}_t$ from previous steps can be stored in an external memory, and the allowable size of the memory is up to $\mathcal{B}_{\mathcal{M}}$.
Typically, samples are drawn from memory, concatenated with the incoming batch and then used for training the segmentation model $\mathcal{S}_t$.
Our goal is to learn an updated segmentation model $\mathcal{S}_t$ to incorporate the novel semantic classes in $\mathcal{C}_t$, and to achieve strong performance on a test data $\mathcal{D}^{ts}_t$. Below we refer to the model learning at each incremental step as a \textit{task}.
We note that $\mathcal{D}^{ts}_t$ includes all the semantic classes up to the $t$-th incremental step, denoted as $\mathcal{Y}_t=\mathcal{Y}_0\cup\left(\cup_{\tau=1}^t\mathcal{C}_\tau\right)$, and the corresponding \textit{background class} $U_t$

\begin{figure*}[t]
	\begin{minipage}{0.47\linewidth}
		\centering

		\begin{algorithm}[H]
			\caption{The Method Overview.}
			\hspace*{-11.2em} {\bf Initialize:} Model $\mathcal{S}_0$; Memory $\mathcal{M}=\emptyset$
			\begin{adjustwidth}{-1em}{}
				\begin{algorithmic}
					\For{$(\mathbf{x}_i, \mathbf{y}_i, \mathcal{R}_i) \in \mathcal{D}^{tr}_0$}
						\State CBES($\mathcal{M}$, $\mathbf{x}_i$, $\mathbf{y}_i$, $\mathcal{R}_i$)
					\EndFor
					\For{\texttt{$t=1,2,3...$}}
					\While{Batch $\mathcal{N}^k_t$ comes}

					\State $\tilde{\mathcal{N}}^k_t = \mathcal{N}_t^k$
					\For{\texttt{$j=1,2,3...,|\mathcal{N}_t^k|$}}
					\State Sample the class $c_{k}$ according to $P_s(c)$ (Eq. \ref{eq:3})
					\thickmuskip=0.5\thickmuskip
					\State Denote $D_s\!=\!\{(\mathbf{x}_l,\mathbf{y}_l,\mathcal{R}_l)\in \mathcal{M}_{t}|\,\exists j\in \mathcal{R}_l,\,\mathbf{y}_{l,j}=c_k\}$
					\State Uniformly sample $(\mathbf{x}_i,\mathbf{y}_i,\mathcal{R}_i)$ from $D_s$
					\State $\tilde{\mathcal{N}}^k_t = \tilde{\mathcal{N}}^k_t \cup \{\mathbf{x}_i,\mathbf{y}_i,\mathcal{R}_i)\}$
					\EndFor
					\State Refine label $\mathbf{y}_i$ in batch $\tilde{\mathcal{N}}_t^k$ by relabeling (Eq. \ref{eq:4})

					\State Update model $\mathcal{S}$ with the loss $\mathcal{L}$ (Eq. \ref{eq:5})
					\State Update class sampling probability $P_s(c)$ (Eq. \ref{eq:3})

					\State $\mathcal{M}^{k+1}_t \leftarrow \mathcal{M}^{k}_t$ \textcolor{gray}{// Update memory}
					\For{$(\mathbf{x}_i, \mathbf{y}_i, \mathcal{R}_i) \in \mathcal{N}^{k}_t$}
					\State $\mathcal{M}^{k+1}_t$$=$$\text{CBES}(\mathcal{M}^{k+1}_{t}, \mathbf{x}_i, \mathbf{y}_i, \mathcal{R}_i)$
					\EndFor
					\EndWhile
					\State $\mathcal{M}_{t+1}^1 = \mathcal{M}_t^{k+1}$
					\EndFor
				\end{algorithmic}
			\end{adjustwidth}
			\label{alg:overview}
		\end{algorithm}
	\end{minipage}
	\hspace{5mm}
	\begin{minipage}{0.47\linewidth}
		\vspace{-12mm}

		\centering

		\begin{algorithm}[H]
			\caption{Class-balanced Exemplar Selection(CBES).}
			\hspace*{-1.50in} {\bf Input:} Memory $\mathcal{M}_t$, data $(\mathbf{x}_i,\mathbf{y}_i,\mathcal{R}_i)$
			\begin{adjustwidth}{-0.9em}{}
			\begin{algorithmic}

				\State Category Set $Y_i = \text{Unique}(\mathbf{y}_i) - \{U_t\}$
				\State $c_{\text{min}} = \arg \min_{c \in Y_i} |\mathcal{M}^{c}_t|$

				\If{ $|\mathcal{M}^{c_{\text{min}}}_t| < \mathcal{B}_{\mathcal{M}}/|\mathcal{Y}_t|$ or $|\mathcal{M}_t| < \mathcal{B}_{\mathcal{M}}$}
						\If{ $|\mathcal{M}_t| == \mathcal{B}_{\mathcal{M}}$}
						\State $c_{\text{max}} = \arg \max_{c \in \mathcal{Y}_t} |\mathcal{M}^{c}_t|$
						\State Randomly remove an image from $\mathcal{M}^{c_{\text{max}}}_t$
						\EndIf
						\State Add $(\mathbf{x}_i, \mathbf{y}_i, \mathcal{R}_i)$ into the memory $\mathcal{M}^{c_{\text{min}}}_t$
				\Else
						\For{$c$ in $Y_i$}
						\State $m_{c}$ $\leftarrow$ the number of saved images of class $c$
						\State $n_{c}$ $\leftarrow$ the number of seen images of class $c$
						\State Sample $p\sim Uniform(0,1)$
						\If{$p \le m_{c}/n_{c}$}
							\State Randomly remove an image from $\mathcal{M}^c_t$
							\State Add $(\mathbf{x}_i, \mathbf{y}_i, \mathcal{R}_i)$ into $\mathcal{M}^c_t$
							\State Break
						\EndIf
						\EndFor
				\EndIf
			\end{algorithmic}
			\end{adjustwidth}
			\label{alg:cbes}
		\end{algorithm}
	\end{minipage}
\end{figure*}

\subsection{Model Architecture}\label{subsec:model_arch}
At each incremental step $t$, we assume a typical semantic segmentation network architecture, which consists of an encoder-decoder backbone network $\mathcal{F}_t$ extracting a dense feature map and a convolution head $\mathcal{H}_t$ that produces the segmentation score map.
Concretely, given the image $\mathbf{x}_i$, the segmentation network first computes the convolutional feature map $\mathbf{F}_i\in\mathbb{R}^{H\times W\times D}$ using the backbone network, where $D$ is the number of channels. $\mathbf{F}_i$ is then fed into the $1\times 1$ convolution head $\mathcal{H}_t$, which has a form of linear classifier at each pixel location. Denote the parameters of the backbone network as $\mathbf{W}$, and the parameters of head as $\mathbf{\Psi} \in \mathbb{R}^{D\times(|\mathcal{Y}_t|+1)}$, our network generates its output $\mathbf{\hat{y}}_i$ as follows,
\begin{align}
&\mathbf{F}_i=\mathcal{F}_t(\mathbf{x}_i;\mathbf{W}), \\
&p(\mathbf{\hat{y}}_i|\mathbf{x}_i)=\text{\small{Softmax}}(\mathcal{H}_t(\mathbf{F}_i))=\prod_{j\in\Omega_i}p(\mathbf{\hat{y}}_{ij}|\mathbf{F}_{ij};\mathbf{\Psi})\label{eq:model}
\end{align}
where $\Omega_i$ represents the 2D image plane and $\mathbf{F}_{ij}\in \mathbb{R}^{D}$ for the pixel $j$. Note that our method is agnostic to specific network designs.

\subsection{Incremental Learning with EM}\label{subsec:em_framework}

We now develop an EM learning framework to efficiently train the model $\mathcal{S}_t$ with a limited memory $\mathcal{M}_t$ in an online fashion. To tackle the stability-plasticity dilemma, our strategy integrates dynamic sampling and pixel relabeling, which enables us to efficiently re-use the training data for fast model adaptation and cope with background drift caused by the partial annotation.

Specifically, we formulate the online learning at each step as a problem of learning with latent variables and develop a stochastic EM algorithm to iteratively update the model parameters.
In each step $t$, our learning procedure sequentially takes incoming mini-batch $\mathcal{N}_t^k$ and updates the model (denoted as $\mathcal{S}^k_t$) as well as the memory (denoted as $\mathcal{M}_t^k$). For each mini-batch, our EM algorithm iterates through three stages: 1) dynamic sampling of training data; 2) an E-step that relabels pixels with missing or background annotations, and 3) an M-step that updates the model parameters with one step of SGD.
Below we denote the model parameter at min-batch $k$ in step $t$ as $\bm{\theta}^k_t=\{\mathbf{W}_t^k, \mathbf{\Psi}_t^k\}$ and describe the details of the three stages in each EM iteration. A complete overview of our algorithm is shown in Algorithm~\ref{alg:overview}.

\paragraph{Dynamic sampling}
We first use a rehearsal strategy~\cite{chaudhry2019tiny} to retrieve a sample of data from the memory, which has the same size as $\mathcal{N}_{t}^{k}$, and add them into the current mini-batch to build the training batch $\tilde{\mathcal{N}}_t^k$ for the subsequent EM update.
Our goal is to achieve efficient model update that learns the new classes and also to retain old ones. To this end, we develop a dynamic sampling strategy to select informative samples from the memory.

Specifically, we devise a category-level sampling strategy that attempts to balance the training samples from the novel classes to be learned and the old ones prone to model forgetting. To achieve this, our method maintains a  \textit{class sampling probability} $P_s(c)$ for $\forall c\in \mathcal{Y}_t$, which will be described below. In order to sample a training image, we first sample a class label $c$ according to $P_s$, and then uniformly sample an image $\mathbf{x}_i$ that has pixels labeled with $c$.
Such a sampling process allows the model to visit the data of certain classes more frequently and to improve the learning of those classes.

Our key design is to dynamically update the class probability $P_s$ according to the prediction confidence in the outputs of current model $\mathcal{S}^k_t$.
To this end, we first compute an estimate of class confidences, denoted as $\{E(c),c\in\mathcal{Y}_t\}$, at every iteration of EM. Concretely, we use a moving average to collect the statistics from batches $\tilde{\mathcal{N}}_t^k$ as follows,
\begin{align}
E(c) &= \mu E_{{old}}(c) + (1-\mu)E_{mb}(c), \quad  \\
E_{mb}(c) &= \underset{{i\in \tilde{\mathcal{N}}^k_t, \mathbf{{y}}_{i,j}=c}}{\text{Mean}}\{p(\mathbf{\hat{y}}_{i,j}=c|\mathbf{x}_i;\bm{\theta}_t^{k-1})\}
\end{align}
where $E_{old}$ and $E_{mb}$ denotes the confidence estimate from last step and current mini-batch, respectively. Here $\mu$ is the momentum coefficient, and $\bm{\theta}_t^{k-1}$ is the model parameters after the update of last step.
At the beginning of a new task $t$, we initialize $E(c)$ as $0$ for classes in $\mathcal{C}_t$.

Given the class confidences, we define the sampling probability $P_s$ with a Gibbs distribution, which assigns higher sampling probability to the classes with lower confidence:
\begin{align}
P_s(c) = \frac{e^{-\eta E(c)}}{\sum_{j=1}^{|\mathcal{Y}_t|} e^{-\eta E(j)}} \quad \forall c\in \mathcal{Y}_t\label{eq:3}
\end{align}
where $\eta$ is the hyperparameter to control the peakiness of the distribution.

\paragraph{E-step}

In E-step, we tackle the problem of background concept drift due to the expansion of semantic label space $\mathcal{Y}_t$, and exploit the unlabeled regions in the training data of incremental steps for model adaption. To this end, we treat the labels of the background pixels in $\{\mathcal{D}^{tr}_0\}$ and the unlabeled pixels in $\{\mathcal{D}^{tr}_t\}_{t\geq 1}$ as hidden variables, and use the current model to infer their posterior distribution given the images and semantic annotations. We then use the posterior to fill in the missing annotations for the subsequent model learning in the M-step. By augmenting the training data with those pseudo labels, we aim to further improve the training efficiency and alleviate the problem of inconsistent background annotation.

Specifically, given a training pair $(\mathbf{x}^\tau_i, \mathbf{y}^\tau_i, \mathcal{R}_{i}^{\tau}) \in \tilde{\mathcal{N}}_t^k$, we first compute an approximate posterior of the latent part of $\mathbf{y}^\tau_i$. Here $\tau\in\{0,\cdots,t\}$ indicates the incremental step it comes from. Denote the region with latent labels as $\bar{\mathcal{R}}_i^\tau$, the posterior of its $j$-th pixel can be estimated as
$p(\mathbf{\hat{y}}_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{k-1})$ based on the current network. We then generate pseudo labels for those pixels by choosing the most likely label predictions as follows,
\begin{align}
\mathbf{{y}}^\tau_{i,j} = \underset{\mathbf{\hat{y}}_{i,j}\in\mathcal{\tilde{Y}}_t - \mathcal{C}_\tau}{\arg\max} p(\mathbf{\hat{y}}_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{k-1}), \quad j\in \bar{\mathcal{R}}^i_\tau
\label{eq:4}
\end{align}
where $\mathcal{C}_\tau$ is the class group from step $\tau$. To filter out unreliable estimation, we only keep the pseudo labels of those pixels with confidence higher than a threshold $\delta$, i.e., $p(\mathbf{{y}}^\tau_{i,j}|\mathbf{x}^\tau_i;\bm{\theta}_t^{k-1})>\delta$. We adopt this hard EM to update our model in the next stage.

\paragraph{M-step}

Our M-step employs the mini-batch training data with augmented annotations to update the model parameters. In order to fully utilize the image annotations, we propose a composite loss consisting of two loss terms as follows, \begin{align}
\mathcal{L} = - \sum_{i=1}^{|\tilde{\mathcal{N}}^k_t|} \left(\sum_{j=1}^{|\mathbf{{y}}^\tau_{i}|} \log p(\mathbf{{y}}^{\tau}_{i,j}|\mathbf{x}_i^\tau;\bm{\theta}_t^k)
	+ \gamma\sum_{j=1}^{|\bar{\mathcal{R}}^\tau_i|} \log \sum_{z \in \tilde{\mathcal{Y}}_{t}\backslash \mathcal{C}_\tau} p( z|\mathbf{x}_i^\tau;\bm{\theta}_t^k)\right)
\label{eq:5}
\end{align}
where $p(\cdot)$ is defined as in Equation~\eqref{eq:model} and $\gamma$ is a weighting coefficient. The first term is the standard cross-entropy loss on every pixel with real annotation or pseudo labels, while the second term penalizes the model on classifying the unlabeled pixels at step $\tau$ into label space $\mathcal{C}_\tau$, which encourages the labeling consistent with the partial groundtruth annotations.

\subsection{Class Balancing Strategies}\label{subsec:class_balanced}
The online incremental learning of semantic segmentation typically has to face severe class imbalance in each step, which is caused by the limited mini-batch size and the data sampling. To address this issue, we introduce two strategies into the training of the segmentation network, which are detailed below.

\paragraph{Cosine Normalization} The model output at each pixel $j$ is represented as a probability vector $p(\mathbf{\hat{y}}_{ij}|\mathbf{F}_{ij};\mathbf{\Psi}^k_t)$ with dimension $|\tilde{\mathcal{Y}}_t|$. Here the $m$-th element of the probability is defined as follows
\begin{align}
	p(\mathbf{\hat{y}}^m_{ij}|\mathbf{F}_{ij};\mathbf{\Psi}^k_t) = \frac{e^{\tau d(\mathbf{u}_m , \mathbf{F}_{ij})}}{\sum_{c=1}^{|\tilde{\mathcal{Y}}_t|} e^{\tau d(\mathbf{u}_c , \mathbf{F}_{ij})}}
\end{align}
where $\tau$ is the temperature used to control the sharpness of the softmax distribution, $\mathbf{u}_m\in\mathbb{R}^D$ is the m-th column of classifier weight $\mathbf{\Psi}^k_t$, and $d$ denotes the cosine distance. This enables us to alleviate the influence of biased norm due to class imbalance and produces more balanced scores across categories.
It is worth noting that we are the first to adopt cosine normalization in the online class incremental learning problem, even though it has been used in other problems~\cite{luo2018cosine,qi2018low, zhang2021distribution}.

\paragraph{Class-balanced Exemplar Selection}
We then extend the class-balanced reservoir sampling (CBRS)~\cite{ChrysakisM20cbrs} to the incremental learning of semantic segmentation by taking into account the multiple classes in each image.
Specifically, at the step $t$, we represent the memory for class $c$ as $\mathcal{M}_t^c = \{(\mathbf{x}_i,\mathbf{y}_i, \mathcal{R}_i)|c \in \text{Unique}(\mathbf{y}_i)\backslash\{U_t\}\}$.
Here we ignore the mini-batch index $k$ for notation clarity.
We build a class-balanced memory by maximizing the minimum size of $\mathcal{M}_t^c$, denoted as $\mathcal{M}_t^{c_{\text{min}}}$.
Concretely, we save an image with the class $c_{\text{min}}$ into the memory if  $|\mathcal{M}_t| < \mathcal{B}_{\mathcal{M}}$ or $\mathcal{M}_t^{c_{\text{min}}}$ is below the average size, i.e., $\mathcal{M}^{c_{\text{min}}}_t$ is smaller than $\mathcal{B}_{\mathcal{M}}/|\mathcal{Y}_t|$.
Otherwise, we perform reservoir sampling for all the classes in $\mathbf{y}^t_i$ until an image is selected.
Algorithm~\ref{alg:cbes} describes the details of exemplar selection strategy.

\section{Experiments}\label{sec:exps}

We evaluate our method\footnote{Code is available at \url{https://github.com/Rhyssiyan/Online.Inc.Seg-Pytorch}} on two online incremental learning benchmarks of semantic segmentation, which are built on the PASCAL VOC 2012~\cite{everingham2015pascal} and ADE20K~\cite{zhou2017scene} datasets, respectively.
Below we first introduce our experimental setup in Sec.~\ref{subsec:exp_setup}, followed by reporting our results and analysis on the PASCAL VOC 2012 benchmark in Sec.~\ref{subsec:voc_exp} and ADE20K in Sec.~\ref{subsec:ade_exp}. Finally, we conduct an ablation study on the ADE20K dataset to analyze the contribution of our method components in Sec.~\ref{subsec:abl_study}.

\vspace{-1mm}
\subsection{Experiment Setup}\label{subsec:exp_setup}
We now describe the setup of our experimental evaluation from three aspects, including the learning protocol of deep segmentation networks, baseline methods for comparison and evaluation metrics.

\begin{table*}[t]
	\centering
	\setlength{\tabcolsep}{3pt}
	\caption{Mean IoU on the Pascal-VOC 2012 dataset for different incremental semantic segmentation splits.  $*$ means the original method does not use an exemplar set and we apply the method on both incoming data and memory. The experiments are conducted under memory size $\mathcal{B}_\mathcal{M}=$100.}	\resizebox{0.99\textwidth}{!}{
	\begin{tabular}{l||cc|c||cc|c||cc|c||cc|c||cc|c||cc|c}
		\toprule
		\multirow{3}{*}{Methods}    & \multicolumn{6}{c||}{\textbf{{19-1}}}   & \multicolumn{6}{c||}{{\textbf{15-5}}} & \multicolumn{6}{c}{{\textbf{15-1}}}    \\
		    & \multicolumn{3}{c||}{\bf{Disjoint}}        & \multicolumn{3}{c||}{\bf{Overlapped}}  & \multicolumn{3}{c||}{\bf{Disjoint}}     & \multicolumn{3}{c||}{\bf{Overlapped}}  & \multicolumn{3}{c||}{\textbf{Disjoint}}      & \multicolumn{3}{c}{\textbf{Overlapped}} \\
		 & \it{0-19}  & \it{20}   & \it{all}  & \it{0-19}  & \it{20}   & \it{all}     & \it{0-15}  & \it{16-20}   & \it{all}   & \it{0-15}  & \it{16-20}   & \it{all} & \it{0-15}  & \it{16-20}   & \it{all}  & \it{0-15}  & \it{16-20}   & \it{all}     \\
		\midrule
		{ER~\cite{chaudhry2019tiny} }        & $71.04$ & $23.36$ & $68.77$ & $71.02$ & $15.96$ & $68.39$ & $65.20$ & $28.94$ & $56.57$ & $65.37$ & $27.84$ & $56.44$ & $66.6$ & $24.98$  & $56.69$ & $68.64$ & $15.37$ & $55.95$ \\
		{LwF.MC*~\cite{rebuffi2017icarl} }   & $72.65$ & $27.56$ & $70.51$ & $72.12$ & $24.46$ & $69.85$ & $69.89$ & $36.55$ & $61.95$ & $71.85$ & $36.26$ & $63.38$ & $68.68$ & $30.81$ & $59.66$ & $69.73$ & $30.19$ & $60.32$ \\
		{iCaRL~\cite{rebuffi2017icarl} }     & $58.81$ & $8.73$ & $56.44$ & $60.38$ & $7.63$  & $57.87$ & $60.06$ & $19.71$ & $50.45$ & $59.39$ & $24.97$ & $51.19$ & $61.48$ & $3.23$  & $47.61$ & $63.78$ & $7.35$  & $50.35$ \\
		{AGEM~\cite{chaudhry2018efficient} } & $71.44$ & $10.17$ & $68.52$ & $72.19$ & $9.11$ & $69.19$ & $64.09$ & $16.83$ & $52.83$ & $65.74$ & $16.91$ & $54.12$ & $66.90$ & $19.67$  & $55.66$ & $65.83$ & $20.18$  & $54.96$ \\
		{MIR~\cite{aljundi2019mir} }         & $68.01$ & $17.57$ & $65.61$ & $67.80$ & $13.68$ & $65.22$ & $64.57$ & $29.00$ & $56.10$ & $64.31$ & $28.28$ & $55.73$ & $64.46$ & $3.36$  & $52.41$ & $67.94$ & $4.67$  & $52.87$ \\
		{MiB*~\cite{cermelli2020modeling} }  & $68.01$ & $27.69$ & $69.92$ & $71.97$ & $21.89$ & $69.59$ & $69.62$ & $34.87$ & $61.35$ & $70.25$ & $32.77$ & $61.33$ & $69.37$ & $29.1$ & $59.78$ & $69.45$ & $29.45$  & $59.92$ \\
		{CoRiSeg~\cite{ozdemir2018learn} }   & $72.19$ & $24.10$ & $69.90$ & $72.90$ & $23.16$ & $70.53$ & $70.49$ & $37.43$ & $62.62$ & $73.93$ & $44.37$ & $66.89$ & $70.73$ & $13.35$ & $57.07$ & $72.56$ & $13.32$ & $58.45$ \\
		\midrule
		{Ours }                              & $\bm{73.43}$ & $\bm{39.10}$ & $\bm{71.80}$ & $\bm{73.76}$ & $\bm{43.42}$ & $\bm{72.32}$ & $\bm{73.45}$ & $\bm{47.89}$ & $\bm{67.36}$ & $\bm{75.56}$ & $\bm{49.89}$ & $\bm{69.45}$ & $\bm{73.88}$ & $\bm{37.15}$ & $\bm{65.14}$ & $\bm{75.77}$ & $\bm{40.34}$ &  $\bm{67.33}$ \\
		\bottomrule
	\end{tabular}}
	\label{tab:pascal_voc_splits}
\end{table*}

\begin{table*}[t]
	\centering
	\caption{The performance of various methods with the fixed memory size $\mathcal{B}_{\mathcal{M}} \in \{20, 50, 100\}$, respectively. The experiments are conducted on the 15-1 disjoint split of PASCAL VOC 2012 dataset. $*$ means the original method does not use an exemplar set and we apply the method on both incoming data and memory}
	\resizebox{0.65\textwidth}{!}{
		\footnotesize
		\begin{tabular}{l|cc|cc|cc}
			\toprule[0.3mm]
			\multirow{2}{*}{\textbf{Methods}}         & \multicolumn{2}{c|}{$\mathcal{B}_{\mathcal{M}}=20$}   & \multicolumn{2}{c|}{$\mathcal{B}_{\mathcal{M}}=50$}             & \multicolumn{2}{c}{$\mathcal{B}_{\mathcal{M}}=100$}                                                                                \\
				\cmidrule{2-7}
			& \textbf{imIoU}                     & \textbf{Final mIoU}                                      & \textbf{imIoU}                     & \textbf{Final mIoU}                                       & \textbf{imIoU}                     & \textbf{Final mIoU}                                       \\
			\midrule
		ER~\cite{chaudhry2019tiny}  & $44.75$ & $34.57$ & $55.30$ & $49.45$ & $61.70$ & $56.69$ \\
		LwF.MC*~\cite{rebuffi2017icarl}	 & $54.46$  & $45.91$  & $60.20$ & $53.25$ & $63.74$ & $59.66$ \\
		iCaRL~\cite{rebuffi2017icarl} & $37.93$ & $30.43$  & $49.07$ & $42.74$ & $52.96$ & $47.61$ \\
		AGEM~\cite{chaudhry2018efficient} &  $50.84$  & $43.89$  & $52.80$ & $46.98$ & $54.15$ & $51.01$ \\
		MIR~\cite{aljundi2019mir} & $47.07$ & $37.46$  & $54.83$ & $47.14$ & $61.81$ & $52.41$ \\
		MiB*~\cite{cermelli2020modeling} & $56.98$  & $48.90$  & $61.93$ & $54.34$ & $64.41$ & $59.78$ \\
		CoRiSeg~\cite{ozdemir2018learn} & $54.50$ & $45.42$ & $60.73$ & $52.06$ & $64.59$ & $57.07$ \\
		\midrule
		Ours	 & $\bm{64.64}$  & $\bm{60.49}$  & $\bm{65.57}$ & $\bm{60.14}$ & $\bm{68.06}$ & $\bm{65.14}$ \\
		\bottomrule[0.3mm]
	\end{tabular}}
	\label{tab:pascal_voc_mem_size}
\end{table*}
\vspace{-1mm}
\paragraph{Online class-incremental Learning Protocol}
To evaluate the performance of online incremental learning methods on fast adaptation and catastrophic forgetting, we adopt the following training protocol.
Following the dataset split proposed in~\cite{cermelli2020modeling}, we start from an initial model trained on a set of base classes and divide the remaining classes into different groups, which defines the incremental tasks to be learned sequentially.
The word "online" means data are coming in streaming form, i.e., each data batch can be seen only once.
In addition, it is allowed to save limited history data in a memory buffer of fixed size.

\paragraph{Comparison Methods}
To demonstrate the efficacy of our framework, we choose a set of typical incremental learning methods as our comparisons.
Experience replay (ER)~\cite{chaudhry2019tiny} is used as a basic continual learning strategy.
LWF~\cite{li2017learning} and ICARL~\cite{rebuffi2017icarl} are two widely adopted methods designed for the offline class-incremental classification.
AGEM~\cite{ChaudhryRRE19agem} and MIR~\cite{aljundi2019mir} are the methods for the online class-incremental classification.
MIB~\cite{cermelli2020modeling} and CoRiSeg~\cite{ozdemir2018learn} are two offline incremental learning methods for semantic segmentation but only the latter considers utilizing memory data.

\paragraph{Evaluation Metrics}
We evaluate the model performance at the end of each incremental task.
For each task, we use the mean of class-wise intersection over union (mIoU) as our metric, which includes the background class at each task.
In addition, we take the \textit{incremental mean IoU} (imIoU), which is computed as the average of the mIoU over different tasks, to measure the overall performance of the model over time

\begin{table*}[t]
	\centering
	\setlength{\tabcolsep}{6pt}
	\caption{Mean IoU on the ADE20K dataset for different class incremental learning scenarios. $*$ means the original method does not use an exemplar set and we apply the method on both incoming data and memory.}
	\resizebox{0.99\textwidth}{!}{
	\begin{tabular}{l||cc|c||cccccc|c||ccc|c}
		\toprule
		\multirow{2}{*}{Methods} & \multicolumn{3}{c||}{{\textbf{100-50}}} & \multicolumn{7}{c||}{{\textbf{100-10}}} & \multicolumn{4}{c}{{\textbf{50-50}}} \\
		\cmidrule{2-15}
		   & \textit{0-100} & \textit{101-150} & \textit{all}  & \textit{0-100} & \textit{100-110} & \textit{110-120} & \textit{120-130} & \textit{130-140} & \textit{140-150} & \textit{all}  & \textit{0-50} & \textit{51-100} & \textit{101-150} & \textit{all}  \\ \hline
		ER~\cite{chaudhry2019tiny}        & $33.02$ & $2.71$ & $22.98$ & $32.59$ & $3.48$ & $4.17$ & $0.01$ & $\bm{1.61}$ & $0.08$ & $22.42$ & $31.70$ & $7.57$ & $3.22$ & $14.28$ \\
		LwF.MC*~\cite{rebuffi2017icarl}   & $34.71$ & $3.69$ & $24.44$ & $33.09$ & $4.65$ & $4.14$ & $0.13$ & $1.60$ & $\bm{0.20}$ & $22.84$ & $34.99$ & $9.07$ & $4.14$ & $16.19$  \\
		iCaRL~\cite{rebuffi2017icarl}     & $21.63$ & $0.00$ & $14.46$ & $22.77$ & $0.00$ & $0.00$ & $0.00$ & $0.00$ & $0.01$ & $15.22$ & $28.13$ & $0.00$ & $0.00$ & $9.50$ \\
		AGEM~\cite{chaudhry2018efficient} & $32.68$ & $0.05$ & $21.88$ & $31.50$ & $0.20$ & $1.35$ & $0.00$ & $0.14$ & $0.00$ & $21.18$ & $31.15$ & $2.16$ & $0.00$ & $11.23$ \\
		MIR~\cite{aljundi2019mir}         & $35.61$ & $0.00$ & $23.82$ & $35.31$ & $0.00$ & $0.00$ & $0.00$ & $0.00$ & $0.00$ & $23.62$ & $37.18$ & $0.91$ & $0.00$ & $12.86$ \\
		MiB*~\cite{cermelli2020modeling}  & $34.01$ & $3.22$ & $23.81$ & $32.11$ & $\bm{5.12}$ & $3.71$ & $0.01$ & $1.41$ & $0.13$ & $22.17$ & $33.55$ & $8.61$ & $3.58$ & $15.37$ \\
		CoRiSeg~\cite{ozdemir2018learn}   & $36.58$ & $0.36$ & $24.58$ & $32.41$ & $0.00$ & $0.21$ & $0.00$ & $0.00$ & $0.00$ & $21.69$& $40.20$ & $1.25$ & $0.31$ & $14.10$ \\

		\midrule
		Ours                              & $\bm{36.80}$ & $\bm{5.33}$ & $\bm{26.38}$ & $\bm{36.66}$ & $4.83$ & $\bm{12.87}$ & $\bm{1.25}$ & $0.42$ & $0.06$ & $\bm{25.81}$ & $34.44$ & $\bm{15.22}$ & $\bm{7.15}$ & $\bm{19.04}$ \\
		\bottomrule
	\end{tabular}}
	\label{tab:ade_mem750}
\end{table*}
\subsection{PASCAL VOC 2012}\label{subsec:voc_exp}

\paragraph{Dataset} The PASCAL VOC 2012 dataset~\cite{everingham2015pascal} consists of 10582 images in the training split and 1449 images in the validation split with a total of 21 different classes, including a background class given by the dataset.
We use the validation split as our test set as the original test set is not publicly available.
Following MiB\cite{cermelli2020modeling}, we also evaluate our method on both \textit{disjoint} and \textit{overlapped} setup where disjoint setup means that the images of each task is disjoint, and overlapped means there exist overlapped images between tasks.
Moreover, we test our method on three different splits on both disjoint and overlapped setups.
We begin with a model trained on 15 foreground classes, and the novel classes come incrementally in two streaming patterns, including 5 classes added at once (15-5) and sequentially, 1 at a time (15-1).
Besides, we also start from a model trained on 19 foreground classes and one class is added at once (19-1).
For the evaluation, we build a test set by taking all the images that contains known classes from the validation split.

\begin{figure*}[t]
	\centering
	\includegraphics[width=\textwidth]{imgs/curves/perf_wrt_steps.pdf}
	\caption{The mIoU of various methods with the Deeplab-v3 at different incremental tasks. The left two figures are the experiments on the VOC 2012 benchmark and the right two figures are on the ADE20K benchmark.}
	\label{fig:voc_steps_perf}
\end{figure*}
\paragraph{Implementation Details}
We conduct our experiments based on DeepLab-v3~\cite{chen2017rethinking} with the ResNet101~\cite{he2016deep} as the backbone.
Following the protocol used in online incremental classification~\cite{aljundi2019mir}, we hold out a validation set from the original training data to tune the hyper-parameters.
The hyper-parameters of our method are $\delta=0.8, \gamma=0.5, \eta=1, \mu=0.9, \tau=12$.
At the initial step, we start with the model pretrained on ImageNet and train the model for 60 epochs and adopt the SGD optimizer with batch size 24, momentum 0.9, weight decay 1e-4.
The learning rate decays following the polynomial decay rule with power $0.9$.
For the incremental stage, the incoming data comes as mini-batches with batch size 4.
We concatenate the incoming batch with a same-sized batch(size=4) sampled from memory to form our mini-batch(size=8) for updating the model.
We adopt SGD optimizer with learning rate 0.001 and momentum 0.9.

\paragraph{Quantitative Results}
Tab.~\ref{tab:pascal_voc_splits} summarizes the comparison results on PASCAL VOC 2012 benchmark.
We can see that our method consistently outperforms other methods by a sizable margin at different incremental splits.
Moreover, our method achieves better performance both on the old classes and the novel classes, which demonstrates that our strategy achieves faster adaptation on the streaming setting and is more robust towards catastrophic forgetting.
Specifically, under the disjoint split of 15-1 setting, compared with the best baseline MiB*, we improve the final mIoU from  59.78 to 65.14(+\textbf{5.36}).
We also plot the curves of mIoU at each incremental task for different methods in Fig.~\ref{fig:voc_steps_perf}.
We can see that the gap between our method and baselines increases over time.

\paragraph{Effects of Memory Size} We also conduct extensive experiments on the 15-1 disjoint split to explore the effect of memory size.
The memory size(i.e. number of saved exemplars) ranges from 20 to 100, which is equivalently 1 to 5 exemplars per class on average, indicating different difficulty of the scenario.
As shown in Tab.~\ref{tab:pascal_voc_mem_size}, our method consistently outperforms other methods on three memory sizes, for example, surpassing MiB by $7.66$ imIoU when $\mathcal{B}_\mathcal{M}=20$.

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Ablation Study on 50-50 split of ADE20K dataset. \textbf{D.S.}: dynamic sampling, \textbf{C.R.}: composite loss with relabeling, \textbf{C.N.}:  cosine normalization, \textbf{CBES}: class-balanced exemplar selection.}
		\resizebox{0.35\textwidth}{!}{
			\begin{tabular}{cccc|c}

				\toprule[0.28mm]

				\multicolumn{4}{c}{Components}   & \multirow{2}{*}{imIoU(\%)}  \\
				CBES & C.R. &  C.N. & D.S. &    \\
				\midrule
				\ding{55} &\ding{55} & \ding{55} & \ding{55} & $17.94$  \\

				\checkmark & \ding{55} & \ding{55} & \ding{55} & $18.63$ \\

				\checkmark & \checkmark &  \ding{55} & \ding{55} &  $19.32$  \\
				\checkmark & \checkmark &  \checkmark  & \ding{55} &  $21.18$  \\

				\checkmark & \checkmark & \checkmark & \checkmark & $22.85$ \\

				\bottomrule[0.28mm]
			\end{tabular}
	}
	\label{tab:abl_ade}
	\vspace{-3mm}
\end{table}

\subsection{ADE20K}\label{subsec:ade_exp}
\paragraph{Dataset} ADE20K~\cite{zhou2017scene} is a large dataset for scene segmentation with 151 classes, including a background class given by the dataset.
The dataset contains around 20K training, 2K validation and 3K test images.
We evaluate our algorithm on the validation set as the test set has not been released.
Like MiB~\cite{cermelli2020modeling}, we adopt disjoint setup for ADE20K by spliting training images into disjoint image sets.
Specifically, we begin with a model pretrained on 100 foreground classes and incrementally add the remaining 50 classes at once(100-50), or sequentially, 10 at a time(100-10).
We also start from a model pretrained on 50 foreground classes and then add the remaining 100 classes sequentially, 50 at a time, denoted as 50-50.
Besides, we also start from a model pretrained on 50 foreground classes and fifty classes are added sequentially, denoted as 50-50.
The evaluation is consistent with PASCAL VOC 2012 benchmark, which builds the test set at task $t$ by selecting all images of known classes from validation set.

\paragraph{Implementation Details}
At step 0, we start with the model pretrained on the ImageNet Dataset and train the model for 60 epochs and adopt the SGD optimizer with batch size 24, momentum 0.9, weight decay 1e-4. The learning rate starts with 1e-2 and decays following the polynomial decay rule with power 0.9.
The hyper-parameters of our method are $\delta=0.8, \gamma=0.1, \eta=10, \mu=0.9, \tau=12$.
For each incremental task, we start with constant learning rate 1e-3 , where the batch size is 8, weight decay 0.0001.
Specifically, both the size of incoming mini-batch and the mini-batch retrieved from memory are 4.
The other configurations are kept same as the ones in the experiments of PASCAL VOC 2012.

\begin{figure}[t]
	\centering
	\begin{minipage}{0.5\textwidth}
		\centering
		\includegraphics[width=0.95\textwidth]{imgs/single_visualization.pdf}

	\end{minipage}
	\caption{Visualization of our method, MiB and LwF.MC on ADE20K dataset. Top row shows the image and the change of actual ground truth over steps. The bottom three rows present the prediction of our method, MiB and LwF.MC at different tasks.}\vspace{-2mm}
	\label{fig:visual}
\end{figure}

\paragraph{Quantitative Results}
In Tab.~\ref{tab:ade_mem750}, we compare our method with previous methods on ADE dataset.
It is evident that our approach still consistently surpasses other methods with a large margin.
Specifically, we outperform LWF.MC* with $2.85\%$ mIoU on the 50-50 split.
As shown in Fig.~\ref{fig:voc_steps_perf}, our method achieves more than 3$\%$ imIoU improvement at final task.

\paragraph{Qualitative Comparison}
Fig.~\ref{fig:visual} shows visualization of the results of our method and two other methods. We can see that our method are more robust towards catastrophic forgetting compared with other methods.

\vspace{-1mm}
\subsection{Ablation Study}\label{subsec:abl_study}
In Tab.~\ref{tab:abl_ade}, we conduct a series of ablation studies on ADE20K to evaluate the effect of our model components.
We can see that each component plays a significant role in improving the final performance.
In particular, class-balanced exemplar selection strategy outperforms the baseline ER by $0.69$ on imIoU.
After adding the composite loss, the performance is improved from $18.63$ to $19.32$.
Moreover, the cosine normalization further brings up the  imIoU by $1.55$.
In the end, the dynamic sampling strategy boosts the method from $21.18$ to $22.85(+\bm{1.67})$.

\section{Related Work}

\paragraph{Semantic Segmentation}
Recent research has made great progress in semantic segmentation based on deep convolutional network~\cite{long2015fully, ronneberger2015u, zhao2017pyramid, ChenPKMY18deeplab}.
However, existing literature typically assumes that the semantic classes of interest and their annotated data are given in advance, which may not be feasible in practical open-world scenarios.
To address this limitation, several recent works start to explore the problem of class-incremental semantic segmentation~\cite{ozdemir2018learn,cermelli2020modeling,michieli2019incremental,tasar2019incremental}.
In particular, ILT~\cite{michieli2019incremental} uses annotations of both novel and old classes for each incremental task and adopts knowledge distillation to alleviate forgetting.
MiB~\cite{cermelli2020modeling} revises the cross-entropy loss and knowledge distillation loss~\cite{hinton15distill} for solving forgetting and the bias caused by the background shift problem.
Notably, MiB introduces a more natural setting requiring only the pixel annotations of novel classes, which is also adopted by our work.

It is worth noting that most existing approaches focus on tackling the incremental semantic segmentation with no memory of historical training data and in an offline manner~\cite{michieli2019incremental,cermelli2020modeling}.
The only exception is the work CoRiSeg~\cite{ozdemir2018learn} for 3D medical image segmentation, which mainly addresses catastrophic forgetting with a distillation-based learning strategy on memory.
In detail, it adopts a confidence-based strategy to select stored exemplars for rehearsal in the future.
In this work, we also consider the more practical setting as in most of incremental classification works~\cite{rebuffi2017icarl,wu2019large,yan2021dynamically} that provides a limited storage for past data.
Nevertheless, our work do not focus on the offline training for each incremental task, and aims to continuously learn the segmentation model from a non-stationary data stream in an online manner.
This enables many applications deployed on mobile devices to fast learn new concepts.

\paragraph{Class Incremental Learning}
Class incremental learning aims to learn novel concepts continuously.
There are mainly three kinds of methods in literature to solve the class incremental learning, which are regularization-based methods, distillation-based methods and structure-based methods.
Regularization-based methods~\cite{kirkpatrick2017overcoming,zenke2017continual} penalizes the change of learned parameters which are important for previously observed classes.
Distillation-based methods~\cite{castro2018end,rebuffi2017icarl,wu2019large,douillard2020podnet} retains learned knowledge via preserving the network output on the saved exemplars with knowledge distillation.
Structured-based methods\cite{Rajasegaran19rpsnet, yan2021dynamically,abati2020conditional} separates parameters learned at different steps from each other to avoid undesirable overlapping in representations.
For the online incremental learning setting,  most existing methods rely on the rehearsal strategy and focus on better utilization of the memory. Specifically,
AGEM~\cite{ChaudhryRRE19agem}, which is an efficient version of GEM~\cite{lopez2017gem}, projects the gradients computed on the new data to the direction that cannot increase the loss on the data sampled from memory.
MIR~\cite{aljundi2019mir} proposes a sampling strategy selecting the maximally interfered data from memory to be learned with new data.
GSS~\cite{aljundi2019gss} focuses on the construction of memory, which aims at maximizing the sample diversity based on their gradient directions.
CBRS~\cite{ChrysakisM20cbrs} develops an class-balancing sampling strategy to deal with class-imbalanced stream of images.
However, CBRS is designed for image classification and cannot be directly applied to semantic segmentation, which typically has instances from multiple classes in each image.
In this work, we propose a new class-balancing strategy to handle this problem.

\paragraph{Re-labeling Strategy}
Re-labeling, also known as self-training \cite{Yarowsky95self,papandreou2015weakly,hung2018adversarial}, is a strategy to learn from partially labeled or unlabeled data. It takes the most probable predictions on unlabeled data as pseudo labels for subsequent model training.
In this work, we incorporate the idea of re-labeling into a unified EM framework in order to accommodate the background concept drift and fill in missing labels of pixels in incoming data.

redpajama_set_name: RedPajamaArXiv


Welcome to Gr8Limo, the premier Lake Elsinore limo rental company. Whether you're looking for LAX limo service or executive chauffeured transportation to Lake Elsinore, let us take you there in style. Our drivers are located throughout Los Angeles County, San Bernardino County, Orange County, Ventura County, Santa Barbara, Riverside County, Palm Springs, San Diego County, and beyond. Explore our site to learn more about Limo Services from Gr8limo.
LAX Airport Limo to Lake Elsinore.
If you're traveling to or from Los Angeles International Airport, we invite you to experience the difference of LAX limo service from Gr8Limo. Before your plane touches down, our advanced monitoring system tracks any last minute changes in your flight. You're not ought to give us a call with regards to your flight status shifts. If you arrive later or earlier than your scheduled arrival time, our uniformed chauffeurs will be there to meet you.
Executive Lincoln Town Car Service from Lake Elsinore to LAX.
The Lincoln Town Car is luxury and most affordable vehicle for LAX Airport Transfers to Lake Elsinore for up to 4 passengers. With its spacious trunk, there is always room for your luggage. It is the first vehicle in automotive history to receive a five-star rating in all five categories. An extended 6" wheel base provides plenty of rear seat leg room amidst a well appointed interior of soft leather trim and satin nickel accents, providing unprecedented comfort and luxury while you travel.
Escalade SUV Service LAX to Lake Elsinore.
Corporate LAX Car Service to Lake Elsinore.
If business brings you to Riverside County, leave the driving to us. All new Lincoln Town Car (MKT), Mercedes Benz S550 sedans, and Luxury SUV's are available at low hourly and daily rates for transportation throughout Southern California. Our Lake Elsinore car service is ideal for executives traveling to meetings or road shows.


Garage cabinets are required to store different instruments, car accessories, equipment, and many other items. They are easy to build, and are extremely useful as they help clear clutter.... Join wall cabinets by driving 1 1⁄4" panhead screws through one cabinet side and into the adjoining cabinet side. Clamp the cabinets together first to make sure the fronts and tops stay flush. Clamp the cabinets together first to make sure the fronts and tops stay flush.
How to build garage cabinets is actually a lot easier than you may think. There are some great do it yourself kits that you can purchase that make building the cabinets and installing them a breeze. But if you decide not to buy a kit, preferring to use blueprints and buying the material yourself. As... Join wall cabinets by driving 1 1⁄4" panhead screws through one cabinet side and into the adjoining cabinet side. Clamp the cabinets together first to make sure the fronts and tops stay flush. Clamp the cabinets together first to make sure the fronts and tops stay flush.
Of the purchase price. Steps sunday build a functional and stylish garage plans with these helpful ideas diy garage storage shed moving service royal storage has evolved and home addition ideas and fresh ideas barn plans free plans to builtin shelving and workspace needs.
Join wall cabinets by driving 1 1⁄4" panhead screws through one cabinet side and into the adjoining cabinet side. Clamp the cabinets together first to make sure the fronts and tops stay flush. Clamp the cabinets together first to make sure the fronts and tops stay flush.


Is Mingle Cash A Scam? – Where Are The Cash?
How does earning at least $57 a day sounds to you by using the Mingle Cash system? If you are looking to earn some extra income you will find this as a good way of giving your income source a boost. However, you might be wondering whether this kind of a scheme will really work and can realistically earn you this amount on autopilot. Mingle Cash is a relatively new program that claims it can help you earn money by offering realistic opportunities to boost your income without really spending for an investment. Is Mingle Cash a scam, you will probably ask. Let us see what we can find out from this program and see whether it is worth your time and effort.
Mingle Cash is just another website that offers income opportunities by offering different ways to earn through a get paid to do system. It claims that you can possibly earn $57 a day from using its system and this is not the maximum amount that you can earn along the process. The best part about its program is that it is free and it boasts how it can provide you passive income opportunities without the need of cashing out some money as investment. So where do you get the cash from Mingle Cash? Its website claims that you can do this by simply surfing the web, do simple tasks and making some referrals.
As a get paid to do system, Mingle Cash will help you earn some money by completing tasks like viewing ads, playing games, watching videos, do some searches on Google, download apps, completing surveys, referring others to use the system and completing offers. Different tasks will earn you different rates as payment and do not keep your hopes high on the amount since they usually pay very cheap. You can possibly earn $0.01 by making Google searches using the Mingle Cash browser. You will be asked to download its browser after signing up a free account from Mingle Cash. By using its browser you will have the chance to do some tasks by seeing different ads like banners, pop ups, video ads and contextual ads. You have control from its settings how often you want these ads to show up from the browser.
You can also earn by playing videos for about 90 seconds. However, you have to watch the video ads that run about 30 minutes before the game starts in order to get credit. The rate for this task depends on the country where you are in with the US and UK having the highest rate of $0.005 for every game played. Other countries pay a measly $0.00005 for every game. By participating in surveys, taking offers and downloading applications you will likely earn more. This is found in the Offer Wall from the site and this program partners with companies like AdgateMedia, Offertoro, AdscendMedia and persona.ly as among many others. These tasks pay from a few cents to $3. The highest paying offer from Mingle Cash is through its referral program where it claims you will earn 5% from your referrals' income by using your referral link. You will also earn additional 5% commission up to 10 downlines.
Becoming a member of Mingle Cash is free so it is quite safe to say there is no harm trying from using this program. An eBook is provided that can help you maximize your earnings by learning how you can promote Mingle Cash more productively. It also has 10 tier referral programs which gives you more opportunity to earn.
Not all ads from Mingle Cash are safe. You will find some tasks asking you to download applications which are marked as fishing URL. Some will be flagged as a program with malware and adware by your antivirus program. There are no definite terms and conditions and privacy page from Mingle Cash. This should make you feel quite uncomfortable since you will be participating on a site without any assurance on the protection it gives to your account and activities. The earnings from Mingle Cash are not reliable or decent enough or worthy of the time and effort that you spend for doing tasks to get paid for.
Is Mingle Cash a scam? It is difficult to say for sure since there are no complaints regarding non-payment from the site. However, there are sufficient reasons to believe that some of its tasks and advertisement are not safe. The pay for the tasks is not something one should look forward to considering you earn very minimal and you need to get more referrals to enjoy commissions. Considering the pennies that you can earn from the tasks it is unlikely that the program is sustainable to keep your referrals do more tasks in the long term which affects your potential income from the program.
Is Paid2YouTube A Scam? – Got Time To Watch Youtube? Is Cash From Home A Scam? Is 7 Day Cash Money a scam? You will be giving away your cash instead. Is Facebook Cash Code A Scam?


text:
layout: page
title: BatchCannotGetBatchFile
number: 204
categories: [AvaTax Error Codes]
disqus: 1
---

## Summary

Batch file could not be found.

## Example

```json
{
  "code": "BatchCannotGetBatchFile",
  "target": "Unknown",
  "details": [
    {
      "code": "BatchCannotGetBatchFile",
      "number": 204,
      "message": "Batch file could not be fetched",
      "description": "The file '-0-' could not be fetched.",
      "faultCode": "Client",
      "helpLink": "http://developer.avalara.com/avatax/errors/BatchCannotGetBatchFile",
      "severity": "Error"
    }
  ]
}
```

## Explanation

AvaTax cannot find the file associated with this batch.

redpajama_set_name: RedPajamaGithub


Trussville (incorporated in 1947) is a city of 20,702 on 33.41 square miles of eastern Jefferson County directly northeast of Birmingham. It adjoins Clay to the west and Argo to the north and is traversed by I-59 and Highway 11. Additionally the northern terminus of I-459 is located within the city. Although not incorporated until 1947, Trussville was established as a settlement in 1820.
Downtown Trussville is located along Highway 11 which is locally Main Street (Trussville) and Gadsden Highway at its intersection with Chalkville Mountain Road and west of its crossing with the Norfolk-Southern Railroad. The Cahaba Homestead Village is located west of downtown between Highway 11 and near I-59.
The first European settler to establish residence in the area was Warren Truss, who entered the area on the Georgia Road with his brothers John and Sam, and constructed a grist mill on the Cahaba River in 1821.
Trussville remained an agricultural community until after the Civil War when the Alabama-Chattanooga Railway was built through the city. By 1886 a blast furnace was built where the current Hewitt-Trussville Middle School now stands.
On June 10, 1947, Trussville was incorporated as a town, and on May 31, 1957, the town officially became a city. It was on this date the City of Trussville was adopted as the official name.
Today Trussville is one of the Birmingham regions most rapidly growing areas. The city has seen much residential and retail construction with two major shopping centers being built within the city during the early 2000s, the Colonial Promenade at Trussville on its western side and both the Colonial Promenade Tutwiler Farm and Pinnacle at Tutwiler Farm along Highway 11 at the I-59/I-459 interchange.
In 2005 Trussville was ranked as number 56 out of 100 in Money magazine's "Best Places to Live".
As of the census of 2010, there were 19,933 people residing in 7,667 housing units in Trussville. The population density was 596.67/mi² with housing units at an average density of 229.5/mi². The racial makeup of the city was 90.3% White and 6.6% African American. 1.25% of the population was Hispanic or Latino of any race.
In 2000 there were 4,588 households out of which 44.2% had children under the age of 18 living with them, 73.3% were married couples living together, 7.6% had a female householder with no husband present, and 16.8% were non-families. 15.3% of all households were made up of individuals and 7.0% had someone living alone who was 65 years of age or older. The average household size was 2.79 and the average family size was 3.11.
In the city the population was spread out with 27.4% under the age of 18, 6.2% from 18 to 24, 28.5% from 25 to 44, 26.1% from 45 to 64, and 11.7% who were 65 years of age or older. The median age was 38.5 years.
The median income for a household in the city was $66,943, and the median income for a family was $71,111. Males had a median income of $48,921 versus $31,806 for females. The per capita income for the city was $27,235. About 2.5% of families and 3.5% of the population were below the poverty line.
The Jefferson County Board of Equalization evaluated 5,337 homes in Trussville and determined that their average market value for 2007 was $227,181, a 2.7% increase since 2006.
Trussville's mayor is Buddy Choat. As of 2016 the five-member Trussville City Council consisted of Alan Taylor, Jef Freeman, Zach Steele, Perry Cook, and Brian Plant.
The city operates its own police and fire departments.
The city is served by Trussville City Schools consisting of Hewitt-Trussville High School, Hewitt-Trussville Middle School, and three elementary schools.
Originally, Paine Intermediate School and Paine Primary School served the elementary student population. In 2016, Paine was split into three neighborhood elementary schools for grades K-5. Paine Elementary School is north of the city on U.S. Highway 11, Magnolia Elementary School is on the eastern edge of the city on Hidden Way Lane, and Cahaba Elementary School is in the central area of the city on Parkway Drive.
This page was last modified on 5 January 2019, at 07:26.


GATES, N.Y. — Gates police say they've cracked the case on not one, not two, but three stolen ATVs and two trucks.
Gates police say 17-year-old Zachary Wallace of Hilton and 18-year-old Jav'ae Hampton of Rochester worked together to steal an off road vehicle from a location on Elmgrove Road last month.
Police say Wallace is also responsible for the burglary of a business and the theft of a truck on Buffalo Road just days before.
While investigating these two incidents police say they recovered not only the stolen truck and ATV, they recovered two other off road vehicles and another truck that had been stolen from other jurisdictions.
Both teens are charged with Grand Larceny.
Wallace is facing an additional charge of burglary.


This will take place on Sunday 9thDecember at the farm, among the animals. The gates open at 3pm with the story starting at 4pm. It will be outside in the fields and the courtyard so wrap up warm and stay on to enjoy festive food and drink afterwards. As always it is free but your donations are very welcome.
If you are able to come along on November 17thfrom 11am onwards to help us make the costumes and props then we would be glad to see you.


Microsoft reported record (and what I consider stunning) revenues for its latest quarter -- giving the company a run rate of $50 billion. So you'd think Wall Street would be popping the corks on their Dom Perignon and lighting up Cohibos. Instead, analysts drilled the stock, boo-hooing that profits fell 28 percent to only $2.6 billion. The profit hit was largely blamed on the Vista delay.


text:
Q: How to Crop UIImage considering screen scale I'm trying to crop a UIImage using a crop box UIView that a user can drag around anywhere in the image view to crop. The logic I'm using to compute the crop rect is as follows:
 extension UIImageView {

    public func computeCropRect(for sourceFrame : CGRect) -> CGRect {

        let widthScale = bounds.size.width / image!.size.width
        let heightScale = bounds.size.height / image!.size.height

        var x : CGFloat = 0
        var y : CGFloat = 0
        var width : CGFloat = 0
        var height : CGFloat = 0
        var offSet : CGFloat = 0

        if widthScale < heightScale {
            offSet = (bounds.size.height - (image!.size.height * widthScale))/2
            x = sourceFrame.origin.x / widthScale
            y = (sourceFrame.origin.y - offSet) / widthScale
            width = sourceFrame.size.width / widthScale
            height = sourceFrame.size.height / widthScale
        } else {
            offSet = (bounds.size.width - (image!.size.width * heightScale))/2
            x = (sourceFrame.origin.x - offSet) / heightScale
            y = sourceFrame.origin.y / heightScale
            width = sourceFrame.size.width / heightScale
            height = sourceFrame.size.height / heightScale
        }

        return CGRect(x: x, y: y, width: width, height: height)
    }
}

The crop box frame looks like this and is positionable anywhere in the frame of the image view by dragging it:

This crop code works just fine until I combine this with another feature I'm trying to support which is the ability to let the user draw using their finger inside of the UIImageView. The code for that looks like this:
  override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {

            if let touchPoint = touches.first {

                let currentPoint = touchPoint.location(in: self)

                 UIGraphicsBeginImageContextWithOptions(frame.size, false, UIScreen.main.scale)

                if let context = UIGraphicsGetCurrentContext() {

                    image?.draw(in: imageEffectsService.computeAspectFitFrameFor(containerSize: frame.size, imageSize: image!.size), blendMode: .normal, alpha: CGFloat(imageOpacity))

                    drawLineAt(startPoint: lastTouchPoint, endPoint: currentPoint, currentContext: context, strokeColor: drawColor)

                    UIGraphicsEndImageContext()
                }
            }
    }

private func drawLineAt(startPoint : CGPoint, endPoint : CGPoint, currentContext : CGContext, strokeColor : UIColor) {

        currentContext.beginPath()
        currentContext.setLineCap(CGLineCap.round)
        currentContext.setLineWidth(brushSize)
        currentContext.setStrokeColor(strokeColor.cgColor)
        currentContext.move(to: startPoint)
        currentContext.addLine(to: endPoint)
        currentContext.strokePath()

        image = UIGraphicsGetImageFromCurrentImageContext()
    }

The crop method loses accuracy once I apply a drawing particularly because of this line:
UIGraphicsBeginImageContextWithOptions(frame.size, false, UIScreen.main.scale)

If instead I use:
UIGraphicsBeginImageContext(frame.size)

My crop code will be accurate but the drawing fidelity will look grainy and low quality because I am not accounting for retina screen devices. My question is how would I modify my crop function to account for the UIScreen.main.scale?

redpajama_set_name: RedPajamaStackExchange


Looking for a unique promotional creation or piece of custom furniture? LVL has the capabilities and connections that can be called upon to create distinctive custom work for your home or business. From marketing and promotional items to furniture and decorative items, we've got you covered. Check out some of our previous work below!
If you've got an idea for a custom project, submit a request for consultation, and let us work with you to help your vision come to life!


text:
GPU=${1:-0}

MIG_ENABLED=$(nvidia-smi --query-gpu=mig.mode.current --format=noheader,csv -i ${GPU})
if [ $MIG_ENABLED == "Enabled" ]
then
echo "Partitioning GPU #$GPU to 7 MIG slices"
sudo nvidia-smi mig -cgi 19,19,19,19,19,19,19 -i $GPU
sudo nvidia-smi mig -cci -i $GPU
else
echo "MIG is not enabled. Please enable MIG on $GPU and rerun"
fi


MAURICIO POCHETTINO has provided an explanation for why Dele Alli could have lashed out at Claudio Yacob.
Alli appeared to land a body blow to Yacob last night as Tottenham played out a frustrating 1-1 draw with West Brom that all but ended their title challenge.
Referee Mike Jones missed the incident but the FA could retrospectively punish the PFA Young Player of the Year.
And that could mean that Alli would play no further part in Spurs' season.
Speaking after the game, Tottenham boss Pochettino has suggested that some players are deliberately provoking Alli as they know he has a temper.
"I hear some comments but I have no time to see the action," said Pochettino of the incident with Yacob.
The disappointing result for Spurs means that Leicester can secure the Premier League title by winning any one of their remaining three games.
But Pochettino refuses to believe that Tottenham are out of the title race just yet.
"It is not over," Pochettino said. "The gap is seven points and we have ahead three games.
"It is true that it is very difficult, more difficult than before the game, but we still believe and we are still fighting.
"We have to hope Leicester drop points and we get three points at Chelsea (next Monday).
"It is important to still fight. The objective is still there. It is more difficult than before but the gap with Arsenal and Manchester City (in fourth and third) is five points and we need to keep this gap.
"It is important to win in case Leicester drop points so we can win the title."


text:
Palmarès
Olimpiadi
Atene 2004: argento nel 2 di coppia.
Pechino 2008: bronzo nel 4 di coppia.
Londra 2008: argento nel 4 di coppia.

Campionati del mondo di canottaggio
2005 - Kaizu: argento nel 4 di coppia.
2006 - Eton: argento nel 2 di coppia.
2007 - Monaco di Baviera: argento nel 4 di coppia.
2010 - Cambridge: bronzo nel 4 di coppia.
2011 - Bled: oro nel 4 di coppia.
2013 - Chungju: oro nel 4 di coppia.

Altri progetti

Collegamenti esterni

redpajama_set_name: RedPajamaWikipedia


text:
Bayern's consistency pays off with ninth straight Bundesliga title
Bayern Munich thrashes Borussia Moenchengladbach 6-0 after being crowned the Bundesliga champion for 2021-22.
MUNICH 09 May, 2021 08:42 IST
Bayern's Thomas Muller, Manuel Neuer and Robert Lewandowski celebrate after the match, post winning the Bundesliga on Saturday. - REUTERS
Bayern Munich on Saturday earned its ninth consecutive Bundesliga title after RB Leipzig's 3-2 loss at Borussia Dortmund and its own later 6-0 demolition of Borussia Moenchengladbach which puts it 10 points clear with two games left to play.
While the Bavarians claimed its 31st German league title and their 30th Bundesliga crown since the introduction of the top division in 1963, it had to work harder for it than in past seasons.
Its six-title haul in 2019/2020 came amid interruptions in play due to the coronavirus pandemic, meaning Bayern has had to dig much deeper this term to stay so dominant in Germany.
By the time the Bundesliga campaign got under way in September it had won five of those titles, with a sixth following in early 2021 with the Club World Cup crown.
But its busy schedule meant Hansi Flick's team never had a summer break to recover from an intense season, with the Champions League final in August followed by the regular season start only weeks later.
This meant it had to play through while rivals Borussia Dortmund and RB Leipzig enjoyed a longer break, fewer international competitions and a much lighter schedule.
Bayern's fatigue started to show after the first few months and a run of eight consecutive league games when they fell behind by at least a goal.
ALSO READ | Sancho double gives Dortmund win over Leipzig, hands Bayern Bundesliga crown
But despite a far from solid backline, the Bavarians lost none of those eight matches, winning five and drawing three.
Its winning cards up front remained top striker Robert Lewandowski who, after taking the world player award for last season, picked up where he left off, and a rejuvenated Thomas Mueller.
Lewandowski has scored 39 times in their 32 league games so far and is within touching distance of Gerd Mueller's all-time record of 40 league goals in one season, set in the 1971/72 campaign.
ALSO READ | Lothar Matthaus trusts Nagelsmann to shine as Bayern coach
For Mueller, who together with David Alaba became the first Bundesliga player to win 10 league crowns, it was a season that could see him earn a Germany callup for the Euros after more than two years, having scored 10 goals and set up another 17.
That consistency proved decisive as title rivals Borussia Dortmund's rollercoaster season eventually put them out of contention, leaving Leipzig as the only real threat to Bayern's enduring domestic dominance.
But a strong finish kept Bayern in front as Leipzig wobbled.
The eventual champions has now won eight of their last 10 league games, while Leipzig managed just five wins in the same period.
As Flick departs after less than two full seasons in charge, to be succeeded by Leipzig's Julian Nagelsmann, he will leave knowing that Bayern has now won more than half of the 58 Bundesliga titles since 1963 - including two under his own short leadership.
1. Werder Bremen 0-0 Bayer Leverkusen
2. Borussia Dortmund 3-2 RB Leipzig
3. VFL Wolfsburg 3-0 FC Union Berlin
Read more stories on Bundesliga.
Hansi Flick
Thomas Muller

redpajama_set_name: RedPajamaCommonCrawl


text:
České filmové nebe (ČFN) – serwis internetowy poświęcony filmom oraz ludziom kina z Czech i Słowacji. Historia bazy sięga 1995 roku, kiedy to powstała jako projekt studencki na Politechnice Ostrawskiej. W 1999 roku twórcy projektu (Radek Vetešník i Petr Herudek) przepisali witrynę do języka PHP i oparli ją na bazie MySQL. W 2002 roku do twórców dołączył Antonín Růžička i powstał fundament serwisu ČFN.

Zobacz też
 Filmweb
 Česko-Slovenská filmová databáze
 KinoPoisk

Przypisy

Linki zewnętrzne
 Oficjalna strona serwisu

Strony internetowe o filmach
Czeskie strony internetowe


text:
Q: Using PHP eval to run code causes Fatal error: Cannot redeclare function I have a PHP script that selects one of many PHP code snippets from the database and executes it using eval. In some cases if two code snippets try to declare a function with the same name, I get the fatal error "cannot redeclare function". Editing the function names in the code snippets is not an option. Is there some way to create a scope or maybe have the functions overwrite each other? Or any other better ideas?
Thanks.
Edit: Loops this code.
ob_start();
try {
    $result = eval($source_code);
} catch(Exception $e) {
    echo "error";
}
$error = ob_get_clean();

A: You have three choices, really.
function_exists()
// this will check for the function's existence before trying to declare it
if(!function_exists('cool_func')){
    function cool_func(){
        echo 'hi';
    }
}

// business as usual
cool_func();

Assign function to a variable
// this will automatically overwrite any uses of $cool_func within the current scope
$cool_func = function(){
    echo 'hi';
}

// call it like this
$cool_func();

Namespacing in PHP >= 5.3.0
/* WARNING: this does not work */
/* eval() operates in the global space */
namespace first {
    eval($source_code);
    cool_func();
}

namespace second {
    eval($source_code);
    cool_func();
}

// like this too
first\cool_func();
second\cool_func();

/* this does work */
namespace first {
    function cool_func(){echo 'hi';}
    cool_func();
}

namespace second {
    function cool_func(){echo 'bye';}
    cool_func();
}

With the second example you would need to eval() the DB code once within every scope which you need to use $cool_func, see below:
eval($source_code);

class some_class{
    public function __construct(){
        $cool_func(); // <- produces error
    }
}

$some_class = new some_class(); // error shown

class another_class{
    public function __construct(){
        eval($source_code); // somehow get DB source code in here :)
        $cool_func(); // works
    }
}

$another_class = new another_class(); // good to go

A: Well, as mentioned by others, you should post the code so we can better help you. But you may want to look into PHP OOP as you can give scope to methods within classes and reference them as such:
ClassOne::myFunction();
ClassTwo::myFunction();

See this for more: http://php.net/manual/en/language.oop5.paamayim-nekudotayim.php


Image Title: Sugar Skull Comforter Sets Comftersugar Set King Throughout Idea 11. Filename: sugar-skull-comforter-sets-comftersugar-set-king-throughout-idea-11.jpg. Image Dimension: 570 x 570 pixels. Images Format: jpg/jpeg. Publisher/Author: Gonzalo Watsica. Uploaded Date: Wednesday - May 09th. 2018 16:36:15 PM. Category: Bedroom. Image Source: freshome.com.
Tap The Thumbnail Bellow to See Related Gallery of "Sugar Skull Comforter Sets Comftersugar Set King Throughout Idea 11"


text:
Hogan Cup final: St Ronan's College out to be history boys
Lurgan school can go to top of the class by landing the All-Ireland senior colleges' title at first attempt
By John Campbell
7 April 2018 9:54 AM
St Ronan's College, Lurgan
When St Ronan's College, Lurgan step out to face Rice College, Westport in the All-Ireland Colleges' Hogan Cup final today, they will be rewriting the history books.
For a school side that made their debut in the Ulster colleges' sector just three years ago and which did not have a pitch to call their own, St Ronan's have made phenomenal progress.
Hogan Cup final LIVE: Updates as St Ronan's, Lurgan face Rice College in All Ireland final at Croke Park
And joint-manager David Wilson, whose impact as a player at club level with St Peter's in Lurgan and with Armagh saw him gain considerable status, now believes that St Ronan's can give under-age football in the Orchard County a boost by winning today as well as thrusting the school onto an unimagined sporting pedestal.
"I suppose you could say that we're being seen as the team that have made it into the Hogan Cup final without a grass pitch of their own but on the other hand it could be said that we have lots of pitches because of the support we have received from all the local clubs," said Wilson.
"They have all rolled in behind us and we in turn like to think that we will be striking a big blow for the future of football in Lurgan and indeed in the county by coming out on top."
With Aghaloo man Mickey Donnelly and Annaclone dynamo Collie Fegan alongside him in the management team, Wilson has helped to nurture a side that stormed to their first MacRory Cup triumph last month and now relish the prospect of being crowned All-Ireland Colleges' champions.
In overcoming St Patrick's College, Armagh in the semi-final and then getting the better of St Mary's GS, Magherafelt in the final, St Ronan's underlined in their triumphant MacRory Cup surge that they have truly come of age.
But they face a huge challenge today against a Rice College side that showed considerable staying power and resolve in overcoming Naas CBS by 0-13 to 1-7 in their semi-final.
In skipper Pat Lambert they have an inspirational player - he scored five second-half points against the Naas side - while Keith Joyce is an outstanding full-forward who could pose big problems for the St Ronan's defence, as indeed could Mark Moran and Stephen Loftus.
But while the Lurgan side acknowledge the pedigree of their opponents, they are strongly focused on their own strategy.
"We have players like Aidan Mulholland, Adam Loughran, the McConvilles Ruairi and Eoin, Jamie Haughey and Oisin Smyth who can help to set the tone for our performance," pointed out Wilson.
"The team has grown in confidence but we are certainly not in the least complacent. This is the biggest game in the careers of our players to date."
They may have lived dangerously on occasions during their MacRory Cup campaign but St Ronan's nonetheless revealed the kind of fortitude, spirit and cohesion that can stand them in good stead today.
They will be facing a Westport side that will take huge support to Headquarters and which is already viewed as providing some of Mayo's most promising future talent.
St Ronan's, though, are unlikely to be fazed by their task this afternoon - their steely determination, strong sense of ambition and inherent togetherness could see them scale a new peak at Croke Park.
Armagh and Tyrone are without key stars in semi-finals mission
Hot-shot Forde can help Tipp the scales in his side's favour
Armagh boss questions added time after 'hard to take' defeat in Mayo
Absence makes Morgan's heart grow fonder in title pursuit
Armagh need to reach another level to beat Mayo, says O'Neill
Lead role: Skipper Jamie Haughey is hoping to lead St Ronan's to glory in the Hogan Cup final


The Bridge Hotel, an Inn in Durham, County Durham.
The Bridge Hotel sits in the heart of Durham City where for over a decade it has offered a wide range of beers, wine, spirits and delicious home cooked food. Situated less than five minutes from both the train and bus stations, the Bridge is ideally located to offer great value accommodation (and our famous home cooked breakfast).
On this site you can find about all the Bridge has to offer, from our food menus to details of the hotel rooms and all the latest news and offers from the pub. You can also contact us directly from the site regarding hotel or table bookings and any other queries you might have. You can keep up with all the latest happenings right here on the site, or by following us on Twitter or Facebook.