text:
\section{Introduction}

A fundamental assumption in machine learning is that training and test data are \gls{IID}. This assumption ensures consistency-results from statistical learning theory, meaning that the learning machine obtained from an \gls{ERM} attains the lowest achievable risk as sample size grows ~\cite{Vapnik98:SLT,scholkopf2019}.
Unfortunately, a considerable amount of research and real-world applications in the past decades has provided a staggering evidence against this assumption \cite{zhao2018,zhao2020,ren2019,taori2020} (see \textcite{damour2020} for case studies).
The violation of the \gls{IID} assumption is usually caused by a \gls{DS} and can result in inconsistent learning machines~\cite{sugiyama2012}, implying the loss of performance guarantee of machine learning models in the real world.
Therefore, to tackle \gls{DS}, recent work advocates for
\gls{DG} \cite{blanchard2011,muandet2013,li2017,li2018,zhou2021}. This generalization to utterly unseen domains is crucial for robust deployment of the models in practical application, especially when new, unforeseeable domains emerge after model deployment. However, the most important question that \gls{DG} seeks to answer is how to identify the right \emph{invariance} allowing for generalization.

The contribution of this work is twofold. First, we advocate that real-world distributions are composed of smaller ``units'' called \emph{invariant elementary distributions} that remain invariant across different domains; see Section \ref{sec:ied} for a motivating example.
Second, we propose to implement this hypothesis through so-called \glspl{GDU}. Specifically, we developed a modular neural network layer that consists of \glspl{GDU}. Each \gls{GDU} learns an embedding of an individual elementary domain that allows us to express the domain similarities during training. For this purpose, we adopt the theoretical framework of \gls{RKHS} to retrieve a geometrical representation of each distribution in the form of a \gls{KME} without information loss \cite{Berlinet04:RKHS,Smola07Hilbert,sriperumbudur2010,Muandet17:KME-Review}. This representation accommodates methods based on analytical geometry to measure similarities between distributions. We show that these similarity measures can be learned and utilized to improve the generalization capability of deep learning models to previously unseen domains.

The remainder of this paper is organized as follows: The theoretical framework is laid out in Section~\ref{sec:content} with our modular \gls{DG} layer implementation shown in Section~\ref{sec:layer}. In Section~\ref{sec:related_work}, we outline related work. Our experimental evaluations are presented in Section~\ref{sec:experiments}. Finally, we discuss potential limitations of our approach and future work in Section~\ref{sec:conclusion}.

\begin{figure*}
\center
    \begin{tikzpicture}[scale=0.6]
        \pgfplotsset{scale=1}
        \begin{axis}[name=Ax1,
                      every axis plot post/.append style={mark=none, domain=-1.5:6, samples=100,smooth, thick},
                      axis x line*=bottom,
                      axis y line*=left,
                      ymin=0]
                  \addplot[blue, ultra thick, dotted] {gauss(0,0.5)};
                  \addplot[red, ultra thick, dotted] {gauss(2.5, 1.75)};
                  \addplot[color=orange] {gauss_mix(2.5, 1.75, 0, 0.5, 1, 0.06)};
        \end{axis}
        \begin{axis}[scale=0.8,
            name=Ax2,at={($(Ax1.north east)+(10,0)$)},
            axis background/.style={fill=green!5},
            anchor=north west,

            axis on top=true,
            axis line style={gray},
            axis x line=middle,
            axis y line=middle,
            ymin=-0.5, ymax=0.5,
            xmin=-0.5, xmax=0.5,
            ticks=none
        	]
        \end{axis}
        \begin{axis}[name=Ax3,at={($(Ax2.north east)+(22,0)$)},
              every axis plot post/.append style={mark=none, domain=-6:8, samples=100,smooth, thick},
              axis x line*=bottom,
              axis y line*=left,
              ymin=0,

            anchor=north west]
        \addplot[blue, ultra thick, dotted] {gauss(0, 0.5)};
        \addplot[red, ultra thick, dotted] {gauss(2.5, 1.75)};

        \addplot[color=violet] {gauss_mix3(-2.5, 1.3, 0, 0.8, 2.5, 1.75, 1.5)};
        \end{axis}

        \foreach \Point [count=\S from 1] in { 0.38,  0.21,  0.19,  0.01, -0.07,  0.11, -0.01,  0.08, -0.22,
        -0.03,  0.18,  0.14, -0.14,  0.28,  0.21, -0.01, -0.16,  0.34,
        0.09,  0.39,  0.05, -0.06, -0.01,  0.08,  0.07, -0.45,  0.13,
        0.05, -0.05, -0.01, -0.27, -0.07, -0.24, -0.1 ,  0.28, -0.14,
        0.34, -0.32,  0.12, -0.56, -0.14, -0.12, -0.33, -0.08, -0.1 ,
        0.09,  0.07, -0.12, -0.  , -0.15,  0.18,  0.04,  0.17, -0.01,
        0.34, -0.18, -0.27, -0.2 , -0.01, -0.07, -0.39, -0.35, -0.09}{
        \draw[thin, cyan!100!black!60] (\Point + 1.5, 0) -- (\Point + 1.5, 0.5);
        }

        \node[cyan] at (1.3, -0.8) {$V_1$};
        \node[cyan] at (7.8, 5.2) (DOMAIN1){\textbullet};
        \draw [thin, latex-latex, cyan, dashed] (1.5, 0.5) [bend left] to (DOMAIN1);
        \node[cyan] at (DOMAIN1.north) {$\mu_{V_1}$};

        \foreach \Point [count=\S from 1] in { 0.87, -2.25,  0.71, -0.66, -1.04, -0.15,  0.63,  1.46,  0.29,
        0.23,  0.49, -0.13, -0.57, -1.08,  1.33, -0.41, -0.11, -0.05,
        0.05, -0.73, -0.41,  0.35,  0.  ,  0.17, -1.79, -0.97,  1.  ,
        1.21, -0.33, -0.31, -0.28, -0.63, -0.25,  0.65, -0.  , -0.64,
        0.99,  0.72, -0.39,  1.06, -0.16, -0.53, -1.45, -0.25,  2.08,
        -0.46, -0.25, -0.06, -0.3 ,  1.36, -0.19,  0.02, -0.1 ,  0.42,
        -0.01, -0.24, -0.71,  0.48, -0.71,  0.59,  1.19, -0.21,  0.1 ,
        }{
        \draw[thin, magenta] (\Point + 3.5, 0) -- (\Point + 3.5, 0.5);
        }

        \node[magenta] at (11.5, 2.5) (DOMAIN2){\textbullet};
        \node[magenta] at (DOMAIN2.south) {$\mu_{V_2}$};
        \node[magenta] at (3.3, -0.8){$V_2$};
        \draw [thin, latex-latex, magenta, dashed] (3.3, 0.5) [bend left] to (DOMAIN2);

        \node[black, thin] at (16, 0.7)(XVAL) [above]{$x_i$};
        \draw[thin, black] (16, 0.0) -- (XVAL);
        \node[black] at (11.1, 5.2) (PHIX)[]{\textbullet};
        \node[black] at (PHIX) [above] {$\phi(x_i)$};
        \draw [black, dashed] (XVAL.south) -- (PHIX);

        \draw [darkgray, thick, latex-latex, dotted] (PHIX) -- (DOMAIN1) node[midway,  above]{$\beta_{i1}$};
        \draw [darkgray, thick, latex-latex, dotted] (PHIX) -- (DOMAIN2) node[midway,  left]{$\beta_{i2}$};

        \node[black] at (3.5, -1.8) {\textbf{During training}};
        \node[black] at (17.3, -1.8) {\textbf{During inference}};

        \end{tikzpicture}
    \caption{A visualization of an ``invariant elementary distribution (I.E.D.)'' assumption for domain generalization (DG): the observed data distributions (orange and violet) are composed of the same set of \emph{unobserved} elementary distributions (bule and red) that remain invariant across different domains. Hence, the first challenge during the training phase (left panel) is to extract these elementary distributions from the observed data (orange). The unobserved elementary distributions are represented by the elementary bases $V_1$ and $V_2$ (cyan and pink). The second challenge during the inference phase (right panel) is to create a weighted ensemble of learning machines that utilize the similarities between the embedding of the unseen observation $\phi(x_i)$ and the embeddings of these distributions $\mu_{V_1}$ and $\mu_{V_2}$ in the RKHS $\mathcal{H}$ (green rectangle) as weights $\beta_{i1}$ and $\beta{i_2}$.}
\end{figure*}\label{fig:concept}

\section{Domain Generalization with Invariant Elementary Distributions} \label{sec:content}

Let $\mathcal{X}$ and $\mathcal{Y}$ be the input and output space, with a joint distribution $\mathbb{P}$.
In the multi-source \gls{DG} setting, we are given a set of $D$ labeled source datasets $\lbrace\mathcal{D}^s_{i}\rbrace_{i=1}^{D}$ with $\mathcal{D}^s_{i} \subseteq \mathcal{X}\times \mathcal{Y}$.
Each of the source datasets is assumed to be \gls{IID} generated by a joint distribution $\mathbb{P}_i^{s}$ with support on $\mathcal{X}\times \mathcal{Y}$, henceforth denoted \emph{domain}. The set of probability measures with support on $\mathcal{X}\times \mathcal{Y}$ is denoted by $\mathcal{P}$. In general, we aim to minimize the empirical risk, see Section \ref{sec:reg} for details. Important notation is summarized in Table \ref{tab:notations}.

\subsection{Invariant Elementary Distributions}
\label{sec:ied}

A multi-source dataset $\mathcal{D}^s$ comprises the merged individual source datasets $\lbrace\mathcal{D}^s_{i}\rbrace_{i=1}^{D}$. Similar to \textcite{mansour2012}, \textcite{albuquerque2019}, and \textcite{hoffman2018a}, we assume that the distribution of the source dataset can be described as a convex combination $\mathbb{P}^{s} = \sum^{D}_{i=1} \alpha^s_{i} \mathbb{P}^s_{i}$, where $\alpha^s= (\alpha^s_1, \dots, \alpha^s_D)$ is an element of the probability simplex (i.e., \mbox{$ \alpha^s \in \Delta^{\tiny{D}} := \lbrace \alpha \in \R^D  \,|\, \alpha_i \geq 0 \wedge \sum_{i=1}^{D}\alpha_{i}=1 \rbrace$}). In other words, $\alpha_i$ quantifies the contribution of each individual source domain to the combined source domains.

\begin{wraptable}{r}{0.5\textwidth}
\caption{Important notation}
\begin{tabular}{c l}\toprule
K & number of elementary distributions\\  \midrule
M & number of elementary domain bases\\  \midrule
N & number of basis vectors\\  \midrule
$\mathbb{P}^s$ & combined multi-source distribution\\  \midrule
$\mathbb{P}^s_i$ & $i$-th single-source distribution\\  \midrule
$\mathbb{P}_i$ & $i$-th elementary distribution\\  \midrule
$V_i$ & $i$-th domain basis \\  \midrule
$v_j^i$ & $j$-th vector in $V_i$\\  \midrule
$\alpha^s_i$ & coefficient for $\mathbb{P}^s_i$\\  \midrule
$\alpha_i$ & coefficient for $\mathbb{P}_i$\\  \midrule
$\beta_{ij}$ & coefficient for sample $x_i$ and $\mu_{V_j}$ \\ \bottomrule
\end{tabular}\label{tab:notations}
\vspace{-25pt}
\end{wraptable}

In contrast to \textcite{mansour2009,mansour2012,hoffman2018a}, we generalize their problem descriptions: We express the distribution of each domain as a convex combination of $K$ elementary distributions $\lbrace \mathbb{P}_{i} \rbrace^{K}_{i=1} \subset \mathcal{P}$, meaning that $\mathbb{P}^{s} = \sum^{K}_{i=1} \alpha_{i} \mathbb{P}_{i}$, where $ \alpha \in \Delta^{\tiny{K}}$. We assume the elementary distributions to be invariant across the domains. The advantage is that we can find an invariant subspace at a more elementary level, as opposed to when we consider the source domains as some sort of basis for all unseen domain

\paragraph{Motivating example.} In this work, we postulate the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains. In practice, the question arises if and when elementary domains evolve. Consider the practical case of classifying the outcome of virus infections based on electronic health records collected from multiple sources such as patients, cohorts, and medical centers. Naturally, several factors determining the trajectory such as gender, pre-existing diseases, and virus mutations can change simultaneously across these sources. While, to a certain degree, these common factors remain invariant across individuals, the contribution of each of these factors may differ between individuals. In terms of the assumptions made in our work, we model each of these factors with a corresponding elementary distribution $\mathbb{P}_{i}$. For a previously unseen individual we can then determine the coefficients $\alpha_i^s$ and therewith quantify the contribution of each factor.

\subsection{Kernel Mean Embedding of Distributions}
\label{sec:kme}

In this work, we leverage the \gls{KME} of distributions \cite{Berlinet04:RKHS,Smola07Hilbert,Muandet17:KME-Review} to discover the elementary distributions and evaluate similarities between them.
Let $\mathcal{H}$ be a reproducing kernel Hilbert space (\gls{RKHS}) of real-valued functions on $\X$ with a reproducing kernel $k: \X \times \X \rightarrow \R$ \cite{scholkopf2001}.
The \gls{KME} of a probability measure $ \mathbb{P} \in \mathcal{P}$ in the \gls{RKHS} $\mathcal{H}$ is defined by a mapping $\phi(\mathbb{P}) = \mu_{\mathbb{P}} := \int_{\X} k(\textbf{x}, \cdot ) \, d \mathbb{P}(\textbf{x})$.
We assume that the kernel $k$ is characteristic, i.e., the mapping $\mu_{\mathbb{P}}$ is injective \cite{Fukumizu04:DRS,Sriperumbudur08injectivehilbert}.
Theoretically, this essential assumption ensures that there is no information loss when mapping the distribution into $\mathcal{H}$.
Given the samples $\lbrace x_1, \dots , x_n \rbrace$ generated \gls{IID} from $\mathbb{P}$, $\mu_{\mathbb{P}}$ can be approximated by the empirical \gls{KME} $\hat{\mu}_{\mathbb{P}} =  (1/n)\sum_{i=1}^{n}k(x_i, \cdot) = (1/n)\sum_{i=1}^{n}\phi(x_i)$.
We refer non-expert readers to \textcite{Muandet17:KME-Review} for a thorough review on this topic.

\vspace{-5pt}
\paragraph{Challenges.}
Figure~\ref{fig:concept} depicts two challenges that come with our assumption of elementary distributions.
First, since we do not have access to the samples from the hidden elementary distributions, the elementary \gls{KME} cannot be estimated directly from the samples at hand.
To overcome this challenge, we instead seek a proxy \gls{KME} $\mu_{V_i} := (1/N)\sum_{j=1}^{N} \phi(v_j^{i}) = (1/N)\sum_{j=1}^{N} k(v_j^{i}, \cdot)$ for each elementary \gls{KME} $\mu_{\mathbb{P}_i}$
from a domain basis $V_{i}$, where $V_{i} = \lbrace v_{1}^{i}, \ldots, v_{N}^{i} \rbrace \subseteq \X$ for all $i \in \lbrace 1, \dots, M \rbrace$.
Hence, the \gls{KME} $\mu_{V_i}$ can be interpreted as the \gls{KME} of the empirical probability measure $\hat{\mathbb{P}}_{V_i} = (1/N)\sum_{j=1}^{N} \delta_{v_j^i}$.
Here, we assume that $M=K$.
The sets $V_{i}$ are referred to as \emph{elementary domain basis}.
Intuitively, the elementary domain basis $V_{1},\ldots,V_{M}$ represents each elementary distribution by a set of vectors that mimic samples generated from the corresponding distribution.
In Figure~\ref{fig:concept}, $V_1$ and $V_2$ as well as their mapping in $\mathcal{H}$ visualize this first challenge.

The second challenge points to the objective of learning the unknown similarity between a single sample $x_i$ and an elementary domain $V_j$, which we denote by $\beta_{ij}$. Considering the advantage of \glspl{KME}, that is to tackle this challenge from a geometrical viewpoint, we quantify similarities between \glspl{KME}. For example, in Figure~\ref{fig:concept}, the similarity between $\phi(x_i)$ and $\mu_{V_1}$ and $\mu_{V_2}$ could be quantified as their distance or angle. These similarity coefficients enable the learning machine to represent a convex combination of elementary domain-specific learning machines, commonly known as ensembles.

\section{Domain Generalization Layer} \label{sec:layer}

This section aims to transfer the theoretical ideas presented in Section~\ref{sec:content} into a deep learning framework. For the purpose of implementation, let $x \in \R^{h \times w}$ denote the input data point and $h_{\xi}:\R^{h \times w}\rightarrow \R^{e}$ the \gls{FE} that maps the input into a low-dimensional representation $\Tilde{x} \in \R^e$. Then the prediction layer $g_{\theta}: \R^e \rightarrow \Y$ infers the label $y$. To tackle the \gls{DG} problem, we introduce a layer module called the \glsfirst{GDU}.
A \gls{GDU} consists of three main components: (1) a similarity function $\gamma: \mathcal{H} \times \mathcal{H} \rightarrow \R$ that is the same for all elementary domains, (2) an elementary basis $V_i$ and (3) a learning machine  $f(\Tilde{x}, \theta_i)$ for each elementary domain $i \in \{1, \dots, M \}$.
The architecture of the layer proposed herein is depicted in Figure~\ref{fig:layerscheme}.

\begin{figure}[ht]
    \centering
    \begin{minipage}[h]{.4\textwidth}
    \centering
    \begin{tikzpicture}[x=1.0cm,y=0.4cm, scale=0.6, every node/.style={scale=0.6}]

    \newcommand \shiftKME{-4}
    \newcommand \shiftMATMUL{-1}
    \newcommand \shiftSUM{1.5}
    \newcommand \shiftOUT{3}
    \newcommand \shiftLOSS{5}
    \newcommand \shiftREG{7}

    \newcommand{7}{7}
    \newcommand{9}{9}
    \newcommand{0}{0}
    \newcommand{-2.5}{-2.5}
    \newcommand{-7}{-7}

    \node at (-1.5, -7) [circle, minimum size=1cm, draw, thick](SUM){$\Sigma$};
    \draw (-0.9, 7+1) rectangle (-2.1, 7-1){};
    \node [scale=1.6] at (-1.5, 7) {$\tilde{x}$};
    \draw []  (-1.5, 7-1) -- (-1.5, 9-4) {};

    \foreach \Point [count=\S from 1] in {-5}{
    \draw (\Point-1, 0 cm +0.2cm) node[minimum height=0.8cm,minimum width=0.8cm, fill=green!10, draw=black, thick] (GDU) {$GDU_{\S}$};
    \draw[] (0, 9-4) -- (\Point, 9-4 ){};
    \draw[-]  (\Point, 9-4) -- (\Point, 0 cm +1cm){};

    \node at (\Point-1, 0 cm -0.8cm) [circle, minimum size=0.2cm, draw, thick](GDUOUT){$\beta_{1}$};
    \draw [-] (GDU.south) -- (GDUOUT){};
    \draw[->]  (\Point, 0 cm +1cm) -| (GDU.north){};

    \draw (\Point+1, 0 cm +0.2cm) node[minimum height=0.8cm,minimum width=0.8cm, draw=black, thick] (LM) {$f(\tilde{x}; \theta_{\S})$};
    \node at (\Point+1, 0 cm -0.8cm) [circle, minimum size=0.2cm, draw, thick](LMOUT){$\hat{y}_{1}$};
    \draw [-] (LM.south) -- (LMOUT){};
    \draw[->]  (\Point, 0 cm +1cm) -| (LM.north){};

    \draw (\Point-2, 0 cm +1.5cm) rectangle (\Point +2 , 0 cm -2cm){};
    \node at (\Point, 0 cm -1.5cm) [circle, minimum size=0.6cm, draw, thick](PROD){};
    \draw[fill=black] (\Point, 0 cm -1.5cm) circle (2pt);
    \draw [->] (LMOUT.south) |- (PROD.east){};
    \draw [->] (GDUOUT.south) |- (PROD.west){};

    \draw [->] (PROD.south) |- (SUM){};
    }

    \foreach \Point [count=\S from 1] in {-1.5}{
    \draw[->, dashed]  (\Point, 9-4) -- (\Point, 0 cm +1cm){};
    \node [] at (\Point, 0 cm ) {$\cdots$};
    \draw (\Point, 0 cm -1.5cm ) node[minimum height=0.6cm,minimum width=0.8cm] (DF) {$\cdots$};

    \draw [-, dashed] (DF.south) |- (SUM.north){};
    }
    \newcommand{green!40!black!100}{green!40!black!100}

    \foreach \Point [count=\S from 1] in {2.5}{
    \draw (\Point-1, 0 cm +0.2cm) node[minimum height=0.8cm,minimum width=0.8cm,  fill=green!10, draw=black, thick] (GDU) {$GDU_{M}$};
    \draw[] (0, 9-4) -- (\Point, 9-4 ){};
    \draw[-]  (\Point, 9-4) -- (\Point, 0 cm +1cm){};

    \node at (\Point-1, 0 cm -0.8cm) [circle, minimum size=0.2cm, draw, thick](GDUOUT){$\beta_{M}$};
    \draw [-] (GDU.south) -- (GDUOUT){};
    \draw[->]  (\Point, 0 cm +1cm) -| (GDU.north){};

    \draw (\Point+1, 0 cm +0.2cm) node[minimum height=0.8cm,minimum width=0.8cm, draw=black, thick] (LM) {$f(\tilde{x}; \theta_{M})$};
    \node at (\Point+1, 0 cm -0.8cm) [circle, minimum size=0.2cm, draw, thick](LMOUT){$\hat{y}_{M}$};
    \draw [-] (LM.south) -- (LMOUT){};
    \draw[->]  (\Point, 0 cm +1cm) -| (LM.north){};

    \draw (\Point-2, 0 cm +1.5cm) rectangle (\Point +2 , 0 cm -2cm){};
    \node at (\Point, 0 cm -1.5cm) [circle, minimum size=0.6cm, draw, thick](PROD){};
    \draw[fill=black] (\Point, 0 cm -1.5cm) circle (2pt);
    \draw [->] (LMOUT.south) |- (PROD.east){};
    \draw [->] (GDUOUT.south) |- (PROD.west){};

    \draw [->] (PROD.south) |- (SUM){};
    }
    \draw [->] (SUM.south) -- (-1.5, -7-4) {};

    \end{tikzpicture}
\end{minipage}
\hspace{.12\textwidth}
\begin{minipage}[h]{.4\textwidth}
\centering
    \begin{tikzpicture}[x=1cm,y=0.5cm, scale=0.6, every node/.style={scale=0.8}]
        \newcommand \kmeX{0}
        \newcommand \cellCornerLU{-3}
        \newcommand \cellCornerRU{3}
        \newcommand \cellTopY{-8}
        \newcommand \cellBotY{-15}
        \newcommand \lineShiftU{-2}
        \newcommand \lineShiftL{-5}

        \draw[dashed,  fill=green!10, draw=black] (\cellCornerLU,  \cellTopY ) rectangle (\cellCornerRU,  \cellBotY );

        \draw (\kmeX+1, \cellTopY + \lineShiftU) node[minimum height=1cm,minimum width=2.4cm, draw=black,fill=white](INNERPRODBOX) {};
        \node at (\kmeX+1, \cellTopY + \lineShiftU) {$\gamma(\phi(\Tilde{x}), \mu_{V_{i}})  $};
        \draw (\kmeX-2, \cellTopY + \lineShiftL) node[minimum height=0.8cm,minimum width=0.8cm, draw=black, fill=white](DOMAIN) {\small$V_i$};

        \draw (\kmeX+1, \cellTopY \lineShiftL) node[minimum height=0.8cm,minimum width=0.8cm, draw=black, fill=white](DOMAINBASIS) {\small$\mu_{V_i}$};
        \node at (\kmeX-4.5, \cellTopY + \lineShiftU ) (KMEINPUT) {$\Tilde{x}$};
        \draw[->, left] (DOMAIN.east) -- (DOMAINBASIS.west) node [midway, sloped, above, scale=1.2, color=black] {$\phi$};

        \draw[->, left] (KMEINPUT.east) -- (INNERPRODBOX.west)node [midway, sloped, above, scale=1.2, color=black] {$\phi$};

        \draw[->, left] (DOMAINBASIS.north) -- (INNERPRODBOX.south);

        \draw[->, left] (INNERPRODBOX.east) -- (\kmeX+5.2, \cellTopY + \lineShiftU)  node[midway, sloped, above] {$ \beta_{i}$};
    \end{tikzpicture

\end{minipage}

\caption{Visualization of the \gls{DG} layer (left panel) and its main component, the \gls{GDU} (right panel). The \gls{DG} layer consists of several \glspl{GDU} that represent the elementary distributions. During training, these \glspl{GDU} learn the elementary domain bases $V_1,\ldots,V_M$ that approximate these distributions.}

\label{fig:layerscheme}
\end{figure}

Essentially, the process is as follows: First, the $j$-th \gls{GDU} takes $\Tilde{x}_i$ as an input and yields $\beta_{ij}$ as an output. The \gls{KME} of each domain basis $V_j$ is required in order to apply $\gamma$ to compute similarity between $\Tilde{x}_i$ and $V_j$. These \glspl{KME} are obtained by $ \phi(V_j) := \mu_{V_j} = (1/N) \sum_{k=1}^{N} \phi(v_{k}^{j}) = (1/N) \sum_{k=1}^{N}k(v_{k}^{j}, \cdot)$. The \gls{GDU}, therefore, has the task to allocate coefficients $\beta_{ij}$ for each elementary domain based on a similarity function $\gamma$. The function $\gamma$ outputs the $\beta_{ij} = \gamma(\phi(\Tilde{x}_i), \mu_{V_j})$ coefficients that in turn represent similarities between the \gls{KME} of both, the corresponding domain basis $V_j$ and the input $\Tilde{x}_i$. Theoretically speaking, $\mu_{V_j}$ and the feature mapping $\phi(\Tilde{x}_i)$ are elements of the associated \gls{RKHS} $\mathcal{H}$, which allow us to evaluate similarities of non-linear features in a higher dimensional feature space.

Each \gls{GDU} is then connected to a learning machine $f(\Tilde{x}_i, \theta_j)$ that yields an elementary domain-specific inference. The final prediction of the layer is then an ensemble of these learning machines $g_{\theta}(\Tilde{x}_i) = \sum_{j=1}^{M}\beta_{ij} f(\Tilde{x}_i, \theta_j)$ where $ \theta = (\theta_1, \ldots, \theta_M).$ In Figure~\ref{fig:layerscheme}, we give an overview of how data is processed and information is stored in the \gls{GDU}.

In summary, \glspl{GDU} leverage the \gls{IED} assumption and represent our algorithmic contribution: The elementary domain basis is stored as weights in the layer. Storing information as a weight matrix (i.e., domain memory) provides the computational advantage of allowing for the use of backpropagation to learn the elementary domain basis. Hence, we avoid the dependency on problem-adaptive methods (e.g., domain-adversarial training) and metadata (e.g., domain labels).

\subsection{Domain Similarity Measures}\label{sec:similarity}
For the similarity function $\gamma$, we consider two similarity measures $H(\phi(x), \mu_{V_{i}})$, namely the \gls{CS}~\cite{kim2019} and \gls{MMD}~\cite{borgwardt2006,gretton2012}. To ensure that the resulting coefficients $\beta_i$ lie on the probability simplex, we apply the kernel softmax function~\cite{gao2019} and interpret its output as the similarity between an observation $\Tilde{x}$ and an elementary domain basis $V_i$. We get
\begin{equation}
    \beta_i = \gamma(\phi(\Tilde{x}), \mu_{V_{i}}) = \frac{\exp\!\big(\kappa H(\phi(\Tilde{x}), \mu_{V_{i}}) \big)}{\sum_{k=1}^{M}\exp\!\big(\kappa H(\phi(\Tilde{x}), \mu_{V_{k}}) \big)},
\end{equation}
where $\kappa > 0$ is a positive softness parameter for the kernel softmax. Geometrically speaking, these similarities correspond to the angle and distance of two \glspl{KME} in the \gls{RKHS} $\mathcal{H}$. The function $\phi$ maps the observation $\Tilde{x}$ and domain basis $V_i$ into $\mathcal{H}$ meaning that $ \phi(x) =\mu_{\delta_{x}}$ is the \gls{KME} of a Dirac measure $\delta_x$ and $\phi(V_i) = \mu_{V_i}$.

\paragraph{\gls{CS}.} The \gls{CS} function

    $H(\phi(x), \mu_{V_{i}}) =  \frac{\langle \phi(x), \mu_{V_{i}} \rangle_{\mathcal{H}}}{\| \phi(x) \|_{\mathcal{H}} \| \mu_{V_i} \|_{\mathcal{H}}}\,$
is used as an angle-based similarity.

\paragraph{\gls{MMD}.} We consider the \gls{MMD} for calculating a distance-based similarity measure.
The distance is then given as $\| \phi(x) - \mu_{V_{i}} \|_{\mathcal{H}}$. Subsequently, the similarity function $H$ is the negative \gls{MMD}: $H(\phi(x), \mu_{V_{i}}) = -\| \phi(x) - \mu_{V_{i}} \|_{\mathcal{H}}$. The intuition behind the negative \gls{MMD} is to put higher weights on samples that are closer to the \gls{KME} of an elementary domain basis.

\subsection{Projection-based Generalization}\label{sec:proj}
For classification tasks, we introduce an alternative approach to infer the $\beta_i$ coefficients that is based on the idea of kernel sparse coding \cite{gao2010,gao2013}. Herein the goal is to find an approximated representation of each feature mapping $\phi(x_i)$ using the elements of a dictionary $\lbrace \mu_{V_{j}} \rbrace_{j=1}^M$. This approach allows us to approximate the feature mapping with these elements by $\phi(x_i) \approx \sum_{j=1}^{M}\beta_{ij} \mu_{V_j}$. In contrast to the aforementioned approaches, an elementary domain \gls{KME} $\mu_{V_j}$ does not necessarily represent the \gls{KME} of an elementary distribution $\mu_{\mathbb{P}_j}$. Therefore, we present another approach that aims to find a set $\lbrace \mu_{V_{j}} \rbrace_{j=1}^M$ that permits $\mu_{\mathbb{P}^s}$ to be represented as a linear combination.

Since $\mathbb{P}$ is assumed to be a convex combination of elementary distributions, we can find a linear combination to represent $\mu_{\mathbb{P}^s}$ by the domain \glspl{KME} $\mu_{V_i}$, as long as $ \mu_{\mathbb{P}^s} \in \mathcal{H}_M := \text{span}\lbrace \mu_{V_i} \,|\, i = 1, \dots, M \rbrace$. The \gls{RKHS} $\mathcal{H}_M$ is a subspace of the actual \gls{RKHS} $\mathcal{H}$, which allows us to represent elements of $\mathcal{H}$ at least approximately in the subspace $\mathcal{H}_M$. By keeping the $\mathcal{H}_M$ large, we gain more representative power. To make $\mathcal{H}_M$ as large as possible, we have to ensure its spanning elements are linearly independent or, even better, orthogonal. Orthogonal \glspl{KME} ensure two desirable properties. First, pairwise orthogonal elements in $\mathcal{H}_M$ guarantee no redundancy. Second, having orthogonal elements allows us to make use of the orthogonal projection. This projection geometrically yields the best approximation of $\phi(x)$ in $\mathcal{H}_M$. In other words, we can achieve the best possible approximation of the feature mapping by using its orthogonal components (see\ Proposition~\ref{proposition_normed}). The orthogonal projection is given by
\begin{align}
    \Pi_{\mathcal{H}_M}: \mathcal{H} \rightarrow \mathcal{H}_M, \quad
    \phi(x) \mapsto \sum_{i=1}^{M}\frac{\langle \phi(x), \mu_{V_i} \rangle_{\mathcal{H}}}{\| \mu_{V_i} \|_{\mathcal{H}}^2}\mu_{V_i}.
\end{align}

\begin{proposition} \label{proposition_normed}
For a \gls{KME} $\mu_{\mathbb{P}}$ of a given mixture distribution $\mathbb{P}$ the following holds $\mu_{\mathbb{P}} \in \text{span} \lbrace \mu_{V_i} \,|\, V_{i}, \forall i=1, \dots, M \rbrace$, where $\langle \mu_{V_{i}},  \mu_{V_{j}} \rangle_{\mathcal{H}} = 0,\forall i \neq j$ (i.e., the \gls{KME} of the elementary domains basis are pairwise orthogonal). The value of the function $\sum_{i=1}^{M}\norm{\mu_{\mathbb{P}}-\beta_i\mu_{V_{i}}}^2$
is minimal if the coefficients are set as $\beta_{i}^{*} = \langle \mu_{\mathbb{P}}, \mu_{V_i} \rangle_{\mathcal{H}}/\| \mu_{V_i} \|_{\mathcal{H}}^2$.
\end{proposition}

The Proposition \ref{proposition_normed} can be used to give an approximation of $\mu_{\mathbb{P}}$ by projecting it into $\mathcal{H}_M$, i.e., $\mu_{\mathbb{P}} \approx \sum_{i=1}^{M}\beta_{i} \mu_{V_i}$ where $\beta_i = \langle \mu_{\mathbb{P}}, \mu_{V_i} \rangle_{\mathcal{H}}/\| \mu_{V_i} \|_{\mathcal{H}}^2$.
This best approximation property is the main advantage of our assumption in Proposition \ref{proposition_normed} (i.e., having orthogonal \gls{KME}) and thus a potential advantage of projection-based \gls{DG}. Appendix~\ref{sup:proof_proposition_normed} provides the proof of Proposition~\ref{proposition_normed}.

\subsection{Model Training}\label{sec:reg}

For model training, we adapt the \gls{DA} framework from \textcite{zhuang2021}. Thus, our learning objective function is formalized as $\mathcal{L}(g) +  \lambda_{D} \Omega_{D}(\| g \|_{\mathcal{H}})$.
The goal of the training can be described in terms of the two components of this function.
Consider a batch of training data $\lbrace x_{1}, \dots, x_{b} \rbrace$, where $b$ is the batch size. During training, we minimize the loss function $\mathcal{L}(g)= \frac{1}{b} \sum_{i=1}^{b} \mathcal{L}(\hat{y}_{i}, y_{i}) = \frac{1}{b} \sum_{i=1}^{b} \mathcal{L}(\sum_{j=1}^{M}\gamma(\phi(x_i), \mu_{V_j}) f_{j}(x_{i}), y_{i})$ for an underlying task and the respective batch size. In addition, our objective is that the model learns to distinguish between different domains. Thus, the regularization $\Omega_{D}$ is introduced to control the domain basis. In our case, we require the regularization $\Omega_{D}$ to ensure that the \gls{KME}s of the elementary domain basis are able to represent the \gls{KME}s of the elementary domains. Therefore, we minimize the \gls{MMD} between the feature mappings $\phi(x_i)$ and the associated representation $\sum_{j=1}^{M}  \beta_{ij}\mu_{V_{j}}$. Note that $\beta_{ij} = \gamma(\phi(x_i), \mu_{V_j})$. Hence, the regularization $\Omega_{D} =  \Omega_{\small{D}}^{OLS}$ is defined as $
    \Omega_{D}^{OLS} \big( \| g \|_{\mathcal{H}} \big) = \frac{1}{b}\sum_{i=1}^{b} \| \phi(x_{i}) - \sum_{j=1}^{M}  \beta_{ij}\mu_{V_{j}} \|_{\mathcal{H}}^2
$ (see Appendix~\ref{ap:detailsonreg} for details). The intuition is the objective to represent each feature mapping $\phi(x_i)$ by the domain \gls{KME}s $\mu_{V_i}$. Thus, we try to minimize the \gls{MMD} between the feature map and a combination of $\mu_{V_i}$. The minimum of the stated regularization can be interpreted as the ordinary least square-solution of a regression-problem of $\phi(x_i)$ by the components of $\mathcal{H}_{M}$. In other words, we want to ensure that the basis $V_i$ is contained in feature mappings $\phi(x_i)$.

In the particular case of projection, we want the \gls{KME} of the elementary domain to be orthogonal to ensure high expressive power. For this purpose, the additional term $\Omega^{\perp}_{D}$ will be introduced to ensure the desired orthogonality. Considering a kernel function with $k(x, x) = 1$, orthogonality would require the Gram matrix $K_{ij} = \langle \mu_{V_i},  \mu_{V_j} \rangle_{\mathcal{H}}$ to be close to the identity matrix $I$. There are a variety of methods for regularizing matrices available~\cite{xie2017,bansal2018}. A well-known method to ensure orthogonality is the \gls{SO} regularization $\Omega_{\small{D}}^{\perp} = \lambda \| K - I \|^2_{F}$ ~\cite{bansal2018}. As pointed out by \textcite{bansal2018}, the \gls{SRIP} and \gls{MC} regularization can be a promising alternative for \gls{SO} and thus are additionally implemented in the \gls{DG} layer. Hence, in the case of projection, the regularization is given by $\Omega_{D} \big( \| g \|_{\mathcal{H}} \big) =  \lambda_{OLS} \Omega^{OLS}_{D} \big( \| g \|_{\mathcal{H}} \big) + \lambda_{ORTH} \Omega_{\small{D}}^{\perp} \big( \| g \|_{\mathcal{H}} \big), ~\lambda_{OLS}, \lambda_{ORTH} \geq~0$.

Lastly, sparse coding is an efficient technique to find the least possible basis to recover the data subject to a reconstruction error \cite{olshausen1997}. Several such applications yield strong performances, for example in the field of computer vision \cite{lee2007,yang2009}. Kernel sparse coding transfers the reconstruction problem of sparse coding into $\mathcal{H}$ by using the mapping $\phi$, and, by applying a kernel function, the reconstruction error is quantified as the inner product \cite{gao2010,gao2013}. To ensure sparsity, we apply the $L_{1}$-norm on the coefficients~$\beta$ and add $ \Omega^{L_{1}}_D(\| \gamma \|) := \| \gamma(\phi(x_i), \mu_{V_j}) \|_{1}$ to the regularization term $\Omega_D$ with the corresponding coefficient $\lambda_{L_1}$. Appendix~\ref{ap:vis_DGLayer} gives a visual overview of the model training.
\section{Related Work} \label{sec:related_work}

\Gls{DG}, also known as \gls{OOD} generalization, is among the hardest problems in machine learning~\cite{blanchard2011,muandet2013,Arjovsky19:IRM}.
In contrast, \gls{DA}, which predates \gls{DG} and \gls{OOD} problems, deals with a slightly simpler scenario in which some data from the test distribution are available~\cite{ganin2015}.
Hence, based on the available data, the task is to develop learning machines that transfer knowledge learned in a source domain specifically to the target domain.
Approaches pursued in \gls{DA} can be grouped primarily into (1) discrepancy-based \gls{DA}~\cite{sun2016,peng2018,ben-david2010,fang2020,tzeng2014,long2015,baktashmotlagh2016} (2) adversary-based \gls{DA}~\cite{tzeng2017,liu2016,ganin2015,long2018}, and (3) reconstruction-based \gls{DA}~\cite{bousmalis2016,hoffman2018,kim2017,yi2017,zhu2017,ghifary2014}.
In \gls{DA}, learning the domain-invariant components requires access to unlabeled data from the target domain.
Unlike problems in \gls{DA}, where the observed data from the test domains can be used to find the most appropriate invariant structures \cite{ben-david2010}, the lack thereof in \gls{DG} calls for a postulation of invariant structure that will enable the \gls{OOD} generalization.

To enable generalization to unseen domains without any access to data from them, researchers have made significant progress in the past decade and developed a broad spectrum of methodologies \cite{zhou2021,zhou2021a,li2019,blanchard2011}. For thorough review see, e.g., \textcite{zhou2021,Wang21:DG}.
Existing works can be categorized into methods based on domain-invariant representation learning \cite{muandet2013,li2018,li2018b}, meta-learning \cite{li2018a,balaji2018}, data augmentation \cite{zhou2020}, to name a few.
Another recent stream of research from a causal perspective includes invariant risk minimization~\cite{Arjovsky19:IRM}, invariant causal prediction~\cite{Peters16:ICP}, and causal representation learning~\cite{Schoelkopf21:CRL}.
The overall motivation here is to learn the representation that is robust to domain-specific spurious correlations. In other words, it is postulated that ``causal'' features are the right kind of invariance that will enable \gls{OOD} generalization.
Despite the successful applications, \gls{DG} remains a challenging research gap.

We differentiate our work from existing ones as follow.
First, we postulate the existence of domain-invariant structure at the distributional level rather than at the data representation, which is a common assumption in \gls{DG}.
This is motivated by theoretical results~\cite{mansour2009,hoffman2018a}, stating that a distribution-weighted combination of source hypotheses represents the ideal hypothesis.
Furthermore, our distributional assumption, as we argued in Section \ref{sec:content}, generalizes previous work that proposes to use domain-specific knowledge to tackle the problem of \gls{DG} from a more elementary setting.
For example, approaches such as \textcite{piratla2020,monteiro2021} can be compared to our \glsplural{GDU} as domain-specific predictors, in the special case, where each elementary domain represents a single source domain. However, \glsplural{GDU} do not assume the existence of a single common classifier for all the domains, providing a combination of multiple common classifiers shared between different source domains.

Second, we incorporate the I.E.D. assumption directly into the design of our model architectures as shown in Figure~\ref{fig:layerscheme}.
Designing effective model architectures for \gls{DG} has been largely neglected~\parencite[Sec.\ 4.1]{zhou2020}.
Third, we do not assume access to the domain labels, which can be difficult to obtain in practice~\cite{niu2017}.
As \gls{DG} methods that can deal with the absence of domain labels (e.g., \cite{huang2020, carlucci2019, li2018d}) are still scarce~\parencite[Sec.\ 4.2]{zhou2020},
we provide an easy-to-apply method to achieve \gls{OOD} generalization: \texttt{model.add(DGLayer())} in TensorFlow.

\section{Experiments}
\label{sec:experiments}

The goal of our experiments is to validate the proposed \gls{IED} assumption and the working principle of the \glsplural{GDU}. Further, we conduct a benchmark experiment based on real-world datasets for \gls{DG}. We distinguish two modes of training the \gls{DG} layer: fine tuning (FT), where we extract features using a pre-trained model, and end-to-end training (E2E), where the \gls{FE} and the \gls{DG} layer are jointly trained.~\footnote{All source code is made available on GitHub \url{https://github.com/im-ethz/pub-gdu4dg}.}.

\subsection{Digits Experiment}
We use five publicly available digits image datasets, namely MNIST~\cite{lecun1998a}, MNIST-M~\cite{ganin2015a}, SVHN~\cite{netzer2011}, USPS, and Synthetic Digits (SYN)~\cite{ganin2015a}. For this experiment, we follow the experimental setup of \textcite{peng2019,feng2020,zhang2020,zhao2018}. The task is to classify digits between zero and nine.
Each of these datasets is considered an out-of-training target domain which is inaccessible during training, and the remaining four are the source domains. Details are given in Appendix~\ref{sec:appendix_digits5}. In Table \ref{tab:results_digits} summarizes the classification results for this experiment. Our \gls{DG} layer noticeably improves mean accuracy and decreases the standard deviation in comparison to the \gls{ERM} and \gls{ERM} ensemble baselines, making the results more stable across the ten iterations reported.

\begin{table*}[!h]
    \caption{Results Digits experiment. All experiments were repeated ten times and the mean (standard deviation) accuracy is reported. Best results according to the mean accuracy are highlighted in \textbf{bold}.}
    \label{tab:results_digits}
\begin{center}
\begin{scriptsize}
\begin{sc}

    \begin{tabular}{llccccc}
          &       & \multicolumn{1}{c}{\textbf{MNIST}} & \multicolumn{1}{c}{\textbf{MNIST-M}} & \multicolumn{1}{c}{\textbf{SVHN}} & \multicolumn{1}{c}{\textbf{USPS}} & \multicolumn{1}{c}{\textbf{SYN}}  \\
\toprule
\multirow{2}{*}[-2pt]{\textit{\textbf{ERM}}} & \textit{Single} & \textit{97.98 (0.34)} & \textit{63.00 (3.20)} & \textit{70.18 (2.74)} & \textit{93.70 (1.74)} & \textit{83.62 (1.47)} \\

\cmidrule{2-7}                       & \textit{Ensemble} & \textit{98.21 (0.39)} & \textit{62.87 (1.50)} & \textit{72.01 (3.59)} & \textit{95.16 (0.89)} & \textit{83.80 (1.22)} \\
    \midrule
                                    & CS        & 98.53 (0.16) & 68.55 (0.80) & 78.90 (1.41) & 95.83 (0.50) & 88.39 (0.82) \\
\cmidrule{2-7} \textbf{FT}          & MMD       & 98.60 (0.08) & 68.62 (0.70) & 79.20 (2.01) & \textbf{96.24 (0.71)} & 88.27 (0.41) \\
\cmidrule{2-7}                      & Projection  & 98.57 (0.17) & 68.56 (0.91) & 79.34 (0.72) & 96.24 (0.71) & 88.58 (0.53) \\
    \midrule
                                    & CS        & 98.62 (0.19) & \textbf{69.25 (0.61)} & \textbf{79.42 (1.27)} & 96.17 (0.52) & 87.92 (0.84) \\
\cmidrule{2-7} \textbf{E2E}         & MMD       & 98.58 (0.16) & 69.04 (0.83) & 79.20 (0.90) & 96.00 (0.44) & 88.18 (0.86) \\
\cmidrule{2-7}                      & Projection & \textbf{98.67 (0.12)} & 68.67 (0.98) & 78.56 (1.68) & 96.24 (0.77) & \textbf{88.77 (0.48)} \\
\bottomrule
\end{tabular}
\end{sc}
\end{scriptsize}
\end{center}
\end{table*}

\subsection{Ablation Study}
We chose the digits dataset to analyze each component of our \gls{DG} layer. We (a) discuss heuristics for choosing $\sigma$ and $M$ (Appendix \ref{sec:heuristics}), (b) vary $M$, $N$ on Figure~\ref{fig:MN_ablation}, and the strength of the regularization terms on Figure~\ref{fig:heatmapp_CS}, Figure~\ref{fig:heatmapp_MMD}, and Figure~\ref{fig:heatmapp_proj} to assess the sensitivity of the \gls{DG} layer to the choice of hyper-parameters (Appendix~\ref{sec:ablation_study}), (c) visualize the output of the \gls{FE} (Figure~\ref{fig:exp_digits_tsne}), and (d) interpret the learned elementary domains (Appendix \ref{sec:interpret_elemdom}). For (a), we cluster the output of the \gls{FE} using k-means algorithm with different number of clusters $K$ and set the number of elementary domains $M$ to the $K^{*}$ yielding the best clustering. For $\sigma$, we resort to the median heuristics \cite{Muandet17:KME-Review}. Further, our ablation study (b) reveals stable results across different sets of hyper-parameters. While the layer is not sensitive to the choice of regularization strength, we recommend not to omit the regularization completely although the computational expanses decrease without the orthogonal regularization. As an illustration in (c), we project the output of the \gls{FE} trained with a dense layer (ERM) and with the \gls{DG} layer by t-SNE (t-distributed stochastic neighbor embedding). The \gls{DG}-trained \gls{FE} yields more concentrated and bounded clusters in comparison to the one trained by \gls{ERM}. For (d), the \gls{MMD} heatmap and t-SNE embeddings of the learned elementary and source domains on Figure \ref{fig:vis_elem_domains} indicate that the \gls{DG} captures distributional structures in the dataset.

\subsection{ECG Experiment}

The PhysioNet/Computing in Cardiology Challenge 2020~\cite{pmid33176294,PhysioNet,physionet2} aims at identifying clinical diagnoses from 12-lead ECG recordings coming from 6 different databases. This publicly available pooled dataset contains 43,101 recordings sampled with various sampling frequencies and lengths. Each recording is labeled as having one or more of 24 cardiac abnormalities and, hence, the task is to perform a multi-label binary classification.
For our experiment, we iterate over the databases, taking one at a time as the test domain while utilizing the remaining 5 databases for training.

The performance was measured according to the original PhysioNet challenge score which is defined as a generalized intersection-over-union score where partial credit is assigned to misdiagnoses that result in similar treatments or outcomes. The score is then adjusted for a solution which always selects the normal/majority class and normalized for the perfect solution.  Therefore, the score can have negative values and a best possible score of 1.

Table~\ref{tab:results_ecgs} reports results for the ECG experiments. For this kind of real-world time-series data we observe an improvement in mean score and reduction in standard deviation over the ERM and ERM ensemble baselines across all generalization tasks. We attribute poorer performance for the \emph{PTB} dataset to the fact that it contains considerably longer recordings than other datasets (except for \emph{INCART} which, however, contains only 75 samples) and higher sampling rate (1000Hz vs 500Hz and 257Hz). The negative challenge score for the \emph{PTB-XL} dataset is due to the presence of previously unobserved labels in other datasets as well as a considerably smaller amount of data for training since the \emph{PTB-XL} dataset comprises the majority of all samples (21,837 out of 43,101).
Appendix~\ref{sec:appendix_ecg} provides details for this experiment.

\begingroup
\setlength{\tabcolsep}{0.23em}
\begin{table*}[!h]
    \caption{Results ECG experiment. All experiments were repeated five times and the mean (standard deviation) challenge metric is reported. Higher is better. Best overall results according to the mean challenge metric are highlighted in \textbf{bold}.}
    \label{tab:results_ecgs}

\begin{center}
\begin{scriptsize}
\begin{sc}
    \begin{tabular}{p{16pt}p{34pt}cccccc}
          &       & \multicolumn{1}{c}{\textbf{CPSC}} & \multicolumn{1}{c}{\textbf{CPSC-Extra}} & \multicolumn{1}{c}{\textbf{INCART}} & \multicolumn{1}{c}{\textbf{PTB}} & \multicolumn{1}{c}{\textbf{PTB-XL}} & \multicolumn{1}{c}{\textbf{G12EC}} \\

\toprule
\multirow{2}{*}[-2pt]{\textit{\textbf{ERM}}} & \textit{Single} & \textit{0.0840 (0.0220)} & \textit{0.2715 (0.0270)} & \textit{0.2290 (0.0059)} & \textit{-8.8206 (0.3908)} & \textit{-0.3373 (0.0403)} & \textit{0.2011 (0.0015)} \\
\cmidrule{2-8}
    & \textsc{\textit{Ensemble}} & \textit{0.1699 (0.0346)} & \textit{0.2488 (0.0079)} & \textit{0.2456 (0.0109)} & \textit{-8.9115 (0.1023)} & \textit{-0.4136 (0.0780)} & \textit{0.2079 (0.0161)} \\
\midrule
    & CS   & 0.1830 (0.0061) & 0.2950 (0.0035) & 0.1595 (0.0313) & -8.8802 (0.1069) & -0.1932 (0.0168) & 0.1853 (0.0036) \\
\cmidrule{2-8} {\textbf{FT}} & MMD   & 0.1877 (0.0077) & 0.3011 (0.0035) & 0.2100 (0.0413) & -8.8082 (0.1458) & -0.1567 (0.0211) & 0.1919 (0.0036) \\
\cmidrule{2-8}          & Projection & \textbf{0.1941 (0.0050)} & \textbf{0.3135 (0.0015)} & -0.1041 (0.0015) & -8.8817 (0.0478) & -0.2166 (0.0191) & 0.2409 (0.0042) \\

\midrule
    & CS   & 0.1067 (0.0170) & 0.2866 (0.0146) & \textbf{0.2539 (0.0289)} & -9.2947 (0.3004) & -0.1651 (0.0494) & 0.1927 (0.0080) \\
\cmidrule{2-8} \textbf{E2E} & MMD   & 0.1034 (0.0143) & 0.2834 (0.0228) & 0.2398 (0.0257) & -9.0600 (0.3100) & -0.1433 (0.0293) & 0.1925 (0.0067) \\
\cmidrule{2-8}          & Projection & 0.1411 (0.0269) & 0.2962 (0.0065) & -0.1467 (0.0513) & \textbf{-8.5904 (0.3310)} & \textbf{-0.0178 (0.0291)} & \textbf{0.2947 (0.0117)}   \\
\bottomrule
\end{tabular}
\end{sc}
\end{scriptsize}
\end{center}
\end{table*}
\endgroup

\subsection{WILDS Benchmark}\label{sec:WILDSBench}

To challenge the \gls{IED} assumption and the generalization capabilities of the \gls{DG} layer, we use WILDS, a curated set of real-world experiments for benchmarking \gls{DG} methods \cite{koh2021wilds}.
We consider the following four datasets: \textit{Camelyon17}, \textit{RxRx1}, \textit{iWildCam}, and \textit{FMoW}, which represent the task of real-world \gls{DG}. Appendix \ref{sec:appendix_wilds} provides details on the datasets and experiments.

Following the WILDS benchmarking procedure \cite{koh2021wilds}, we compare our proposed \gls{DG} layer to the following baselines. First, empirical risk minimization (\gls{ERM}), which minimizes the average training loss over the pooled dataset.
Second, a group of \gls{DG} algorithms provided by the WILDS benchmark, namely, Coral, Fish, IRM, and DRO. The Coral algorithm introduces a penalty for differences in means and covariances of the domains feature distributions. The Fish algorithm achieves \gls{DG} by approximating an inter-domain gradient matching objective, i.e., maximizing the inner product between gradients from different domains \cite{shi2021}. Conceptually, Fish learns feature representations that are invariant across domains.
Invariant risk minimization (IRM) introduces a penalty for feature distributions with different optimal classifiers for each domain \cite{Arjovsky19:IRM}. The idea is to enable \gls{OOD} generalization by learning domain-invariant causal predictors.
Lastly, group \gls{DRO} explicitly minimizes the training loss on the worst-case domain \cite{sagawa2020,hu2018}.

We present our benchmarking results in Table \ref{table:wilds}. When reproducing the \gls{ERM} results, we noticed two deviations from the original results presented in \textcite{koh2021wilds}. For \textit{Camelyon17}, we achieved a better accuracy with a simpler \gls{FE}. For the remaining datasets, our \gls{ERM} achieved lower performances although we used the same specifications made in \textcite{koh2021wilds}. Hence, we report our reproduced \gls{ERM} results as a baseline for comparison. As shown by \textcite{koh2021wilds}, \gls{ERM} remains one of the strongest baselines in real-world \gls{DG} problems.
Our method consistently increases the mean performance and decreases the standard deviation in comparison to \gls{ERM}. For \textit{RxRx1}, however, our method does not outperform the baseline.
In cotrast to our digits and ECG experiments, the WILDS benchmarking requires deeper \gls{FE} such as ResNet-50 or DenseNet-121. We discuss the scalability of our \gls{DG} Layer in the Appendix~\ref{sec:appendixScalabilit}.
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
\begin{table*}[!h]
\caption{Results on WILDS benchmarking tasks. We compute the metrics following \textcite{koh2021wilds} and report the mean (standard deviation) across ten runs for Camelyon17 and three runs for the remaining. Best performances in comparison with our \gls{ERM} are highlighted in \textbf{bold}.}
\label{table:wilds}
\begin{center}
\begin{scriptsize}

\begin{sc}
\begin{tabular}{p{16pt}p{34pt}cccccP{40pt}}

 & & \textbf{Camelyon17} & \textbf{RxRx1} & \multicolumn{2}{c}{\textbf{iWildCam}} & \multicolumn{2}{c}{\textbf{FMoW}} \\
      && \multirow{2}{*}{Avg Acc} & \multirow{2}{*}{Avg Acc} & \multirow{2}{*}{Avg Acc}  & \multirow{2}{*}{Macro F1} & \multirow{2}{*}{Avg Acc} & Avg Worst-Region Acc \\
\toprule
\textit{\textbf{ERM}}                                 &                       & \textit{77.8 (12.7)}    & \textit{\textbf{26.9 (0.0)}} &\textit{67.0 (3.9)}&\textit{ 16.7 (0.6)}          & \textit{46.1 (1.5)} & \textit{24.5 (0.9)} \\
\cmidrule{1-8}                                                &CS             & \textbf{79.2 (10.1)}    & 25.3 (0.1) & 68.9 (1.6)  & 16.5 (0.4) & 51.7 (0.2) & 30.4 (0.3)\\
\cmidrule{2-8}\textbf{FT}                                     &MMD            & 78.5 (11.9)             & 25.3 (0.1) & 70.3 (3.2)  & 16.9 (0.4) & \textbf{51.9 (0.7)} &  \textbf{31.8 (0.9)}\\
\cmidrule{2-8}                                                &Projection      & 78.6 (11.9)             & 25.1 (0.3) & \textbf{71.2 (2.6)}  & \textbf{17.2 (0.6)} & 51.5 (0.2) & 30.2 (1.1) \\
\cmidrule{1-8}                                                &CS             & 76.7 (11.4)             & 16.3 (0.2) & 65.6 (1.8)  & 15.6 (1.0) & 47.0 (0.6) &  28.2 (0.7)\\
\cmidrule{2-8}\textbf{E2E}                                    &MMD            & 75.5 (12.8)             & 16.2 (0.1) & 67.5 (2.6)  & 15.7 (0.6) & 47.6 (2.1) &  27.1 (2.7)\\
\cmidrule{2-8}                                                &Projection      & 76.7 (6.3)              & 13.9 (0.1) & 67.4 (0.2)  & 15.4 (1.1)  & 47.4 (0.9) &  28.6 (1.6)\\
\bottomrule \toprule
\multicolumn{8}{c}{\textit{Results reported by WILDS \cite{koh2021wilds}}} \\
\midrule
\multicolumn{2}{l}{\textbf{ERM}}                          & 70.3 (6.4)     & 29.9 (0.4)     &71.6 (2.5)& 31.0 (1.3) & 53.0 (0.6) & 33.7 (1.5)   \\
\multicolumn{2}{l}{\textbf{CORAL}}                              & 59.3 (7.7)     & 28.4 (0.3)     &73.3 (4.3)& 32.8 (0.1) & 50.5 (0.4) & 31.7 (1.2) \\
\multicolumn{2}{l}{\textbf{Fish}}                               & 74.7 (7.1)     & --                 &64.7 (2.6)& 22.0 (1.8) & 51.8 (0.3) & 34.6 (0.2)  \\
\multicolumn{2}{l}{\textbf{IRM}}                                & 64.2 (8.1)     & 8.2 (1.1)      &59.8 (3.7)& 15.1 (4.9) & 50.8 (0.1) & 30.0 (1.4)  \\
\multicolumn{2}{l}{\textbf{Group DRO}}                          & 68.4 (7.3)     & 23.0 (0.3)     &72.7 (2.0)& 23.9 (2.1) & 52.1 (0.5) & 30.8 (0.8)   \\

\bottomrule
\end{tabular}
\end{sc}
\end{scriptsize}
\end{center}
\end{table*}

\section{Conclusion and Discussions}
\label{sec:conclusion}

We introduced the \gls{IED} assumption, postulating that real-world distributions are composed of elementary distributions that remain invariant across different domains. This invariance thus enables knowledge transfer to unseen domains. Empirical results based on real-world data support the validity of the \gls{IED} assumption. In \textit{Camelyon17}, for example, we would expect similar subpopulations across hospitals, which may constitute elementary domains. In contrast, when the number of elementary domains become excessively large, as we suspect in the \textit{RxRx1} dataset, representing them becomes difficult. Further, we presented a modular neural network layer consistsing of Gated Domain Units (GDUs) that leverage the \gls{IED} assumption. Our \glspl{GDU} can substantially improve the downstream performance of learning machines in real-world \gls{DG} tasks.

We expect the \gls{IED} assumption and \glsplural{GDU} to be adapted yielding novel applications that tackle \gls{DG}.
For example, adaptively increasing the number of elementary domains during learning until their distributional variance, as a measure of their heterogeneity, reaches a plateau.

\paragraph{Limitations.} A major limitation of our \gls{IED} assumption is to provide theoretical evidence that this assumption holds in practice. We aim to expand the scope of the theoretical understanding of the \gls{IED} assumption and the \glspl{GDU}.
In addition, the particular theoretical setting of \textcite{albuquerque2019} (i.e., each elementary domain represents a source domain) seems promising to extend their generalization guarantee to  cases where our \gls{IED} assumption holds.
Second, our GDU layer induces additional computational overhead due to the regularization and model size that increases as a function of the number of elementary domains. Noteworthy, our improvement is achieved with a relatively small number of elementary domains indicating that the increased complexity is not a coercive consequence of applying the \gls{DG} layer. Also, the results achieved are not a consequence of increased complexity, as the ensemble baseline shows.

\paragraph{Societal Impact.}
On a downside, our method may be used to target unknown populations or ethnicities with potentially harmful applications. For example, consider facial recognition in public spaces, where our approach can improve model performance for yet unseen populations, thereby allowing for more extensive surveillance as a whole. However, this impact is a general result of \gls{DG}.

\section*{References}

\printbibliography[heading=none]

\newpage

redpajama_set_name: RedPajamaArXiv


text:
#include <sys/cdefs.h>
__FBSDID("$FreeBSD: release/9.1.0/sys/kern/subr_taskqueue.c 225570 2011-09-15 08:42:06Z adrian $");

#include <sys/param.h>
#include <sys/systm.h>
#include <sys/bus.h>
#include <sys/interrupt.h>
#include <sys/kernel.h>
#include <sys/kthread.h>
#include <sys/limits.h>
#include <sys/lock.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/proc.h>
#include <sys/sched.h>
#include <sys/taskqueue.h>
#include <sys/unistd.h>
#include <machine/stdarg.h>

static MALLOC_DEFINE(M_TASKQUEUE, "taskqueue", "Task Queues");
static void	*taskqueue_giant_ih;
static void	*taskqueue_ih;

struct taskqueue_busy {
	struct task	*tb_running;
	TAILQ_ENTRY(taskqueue_busy) tb_link;
};

struct taskqueue {
	STAILQ_HEAD(, task)	tq_queue;
	taskqueue_enqueue_fn	tq_enqueue;
	void			*tq_context;
	TAILQ_HEAD(, taskqueue_busy) tq_active;
	struct mtx		tq_mutex;
	struct thread		**tq_threads;
	int			tq_tcount;
	int			tq_spin;
	int			tq_flags;
	int			tq_callouts;
};

#define	TQ_FLAGS_ACTIVE		(1 << 0)
#define	TQ_FLAGS_BLOCKED	(1 << 1)
#define	TQ_FLAGS_PENDING	(1 << 2)

#define	DT_CALLOUT_ARMED	(1 << 0)

#define	TQ_LOCK(tq)							\
	do {								\
		if ((tq)->tq_spin)					\
			mtx_lock_spin(&(tq)->tq_mutex);			\
		else							\
			mtx_lock(&(tq)->tq_mutex);			\
	} while (0)

#define	TQ_UNLOCK(tq)							\
	do {								\
		if ((tq)->tq_spin)					\
			mtx_unlock_spin(&(tq)->tq_mutex);		\
		else							\
			mtx_unlock(&(tq)->tq_mutex);			\
	} while (0)

void
_timeout_task_init(struct taskqueue *queue, struct timeout_task *timeout_task,
    int priority, task_fn_t func, void *context)
{

	TASK_INIT(&timeout_task->t, priority, func, context);
	callout_init_mtx(&timeout_task->c, &queue->tq_mutex, 0);
	timeout_task->q = queue;
	timeout_task->f = 0;
}

static __inline int
TQ_SLEEP(struct taskqueue *tq, void *p, struct mtx *m, int pri, const char *wm,
    int t)
{
	if (tq->tq_spin)
		return (msleep_spin(p, m, wm, t));
	return (msleep(p, m, pri, wm, t));
}

static struct taskqueue *
_taskqueue_create(const char *name __unused, int mflags,
		 taskqueue_enqueue_fn enqueue, void *context,
		 int mtxflags, const char *mtxname)
{
	struct taskqueue *queue;

	queue = malloc(sizeof(struct taskqueue), M_TASKQUEUE, mflags | M_ZERO);
	if (!queue)
		return NULL;

	STAILQ_INIT(&queue->tq_queue);
	TAILQ_INIT(&queue->tq_active);
	queue->tq_enqueue = enqueue;
	queue->tq_context = context;
	queue->tq_spin = (mtxflags & MTX_SPIN) != 0;
	queue->tq_flags |= TQ_FLAGS_ACTIVE;
	mtx_init(&queue->tq_mutex, mtxname, NULL, mtxflags);

	return queue;
}

struct taskqueue *
taskqueue_create(const char *name, int mflags,
		 taskqueue_enqueue_fn enqueue, void *context)
{
	return _taskqueue_create(name, mflags, enqueue, context,
			MTX_DEF, "taskqueue");
}

/*
 * Signal a taskqueue thread to terminate.
 */
static void
taskqueue_terminate(struct thread **pp, struct taskqueue *tq)
{

	while (tq->tq_tcount > 0 || tq->tq_callouts > 0) {
		wakeup(tq);
		TQ_SLEEP(tq, pp, &tq->tq_mutex, PWAIT, "taskqueue_destroy", 0);
	}
}

void
taskqueue_free(struct taskqueue *queue)
{

	TQ_LOCK(queue);
	queue->tq_flags &= ~TQ_FLAGS_ACTIVE;
	taskqueue_terminate(queue->tq_threads, queue);
	KASSERT(TAILQ_EMPTY(&queue->tq_active), ("Tasks still running?"));
	KASSERT(queue->tq_callouts == 0, ("Armed timeout tasks"));
	mtx_destroy(&queue->tq_mutex);
	free(queue->tq_threads, M_TASKQUEUE);
	free(queue, M_TASKQUEUE);
}

static int
taskqueue_enqueue_locked(struct taskqueue *queue, struct task *task)
{
	struct task *ins;
	struct task *prev;

	/*
	 * Count multiple enqueues.
	 */
	if (task->ta_pending) {
		if (task->ta_pending < USHRT_MAX)
			task->ta_pending++;
		return (0);
	}

	/*
	 * Optimise the case when all tasks have the same priority.
	 */
	prev = STAILQ_LAST(&queue->tq_queue, task, ta_link);
	if (!prev || prev->ta_priority >= task->ta_priority) {
		STAILQ_INSERT_TAIL(&queue->tq_queue, task, ta_link);
	} else {
		prev = NULL;
		for (ins = STAILQ_FIRST(&queue->tq_queue); ins;
		     prev = ins, ins = STAILQ_NEXT(ins, ta_link))
			if (ins->ta_priority < task->ta_priority)
				break;

		if (prev)
			STAILQ_INSERT_AFTER(&queue->tq_queue, prev, task, ta_link);
		else
			STAILQ_INSERT_HEAD(&queue->tq_queue, task, ta_link);
	}

	task->ta_pending = 1;
	if ((queue->tq_flags & TQ_FLAGS_BLOCKED) == 0)
		queue->tq_enqueue(queue->tq_context);
	else
		queue->tq_flags |= TQ_FLAGS_PENDING;

	return (0);
}
int
taskqueue_enqueue(struct taskqueue *queue, struct task *task)
{
	int res;

	TQ_LOCK(queue);
	res = taskqueue_enqueue_locked(queue, task);
	TQ_UNLOCK(queue);

	return (res);
}

static void
taskqueue_timeout_func(void *arg)
{
	struct taskqueue *queue;
	struct timeout_task *timeout_task;

	timeout_task = arg;
	queue = timeout_task->q;
	KASSERT((timeout_task->f & DT_CALLOUT_ARMED) != 0, ("Stray timeout"));
	timeout_task->f &= ~DT_CALLOUT_ARMED;
	queue->tq_callouts--;
	taskqueue_enqueue_locked(timeout_task->q, &timeout_task->t);
}

int
taskqueue_enqueue_timeout(struct taskqueue *queue,
    struct timeout_task *timeout_task, int ticks)
{
	int res;

	TQ_LOCK(queue);
	KASSERT(timeout_task->q == NULL || timeout_task->q == queue,
	    ("Migrated queue"));
	KASSERT(!queue->tq_spin, ("Timeout for spin-queue"));
	timeout_task->q = queue;
	res = timeout_task->t.ta_pending;
	if (ticks == 0) {
		taskqueue_enqueue_locked(queue, &timeout_task->t);
	} else {
		if ((timeout_task->f & DT_CALLOUT_ARMED) != 0) {
			res++;
		} else {
			queue->tq_callouts++;
			timeout_task->f |= DT_CALLOUT_ARMED;
		}
		callout_reset(&timeout_task->c, ticks, taskqueue_timeout_func,
		    timeout_task);
	}
	TQ_UNLOCK(queue);
	return (res);
}

void
taskqueue_block(struct taskqueue *queue)
{

	TQ_LOCK(queue);
	queue->tq_flags |= TQ_FLAGS_BLOCKED;
	TQ_UNLOCK(queue);
}

void
taskqueue_unblock(struct taskqueue *queue)
{

	TQ_LOCK(queue);
	queue->tq_flags &= ~TQ_FLAGS_BLOCKED;
	if (queue->tq_flags & TQ_FLAGS_PENDING) {
		queue->tq_flags &= ~TQ_FLAGS_PENDING;
		queue->tq_enqueue(queue->tq_context);
	}
	TQ_UNLOCK(queue);
}

static void
taskqueue_run_locked(struct taskqueue *queue)
{
	struct taskqueue_busy tb;
	struct task *task;
	int pending;

	mtx_assert(&queue->tq_mutex, MA_OWNED);
	tb.tb_running = NULL;
	TAILQ_INSERT_TAIL(&queue->tq_active, &tb, tb_link);

	while (STAILQ_FIRST(&queue->tq_queue)) {
		/*
		 * Carefully remove the first task from the queue and
		 * zero its pending count.
		 */
		task = STAILQ_FIRST(&queue->tq_queue);
		STAILQ_REMOVE_HEAD(&queue->tq_queue, ta_link);
		pending = task->ta_pending;
		task->ta_pending = 0;
		tb.tb_running = task;
		TQ_UNLOCK(queue);

		task->ta_func(task->ta_context, pending);

		TQ_LOCK(queue);
		tb.tb_running = NULL;
		wakeup(task);
	}
	TAILQ_REMOVE(&queue->tq_active, &tb, tb_link);
}

void
taskqueue_run(struct taskqueue *queue)
{

	TQ_LOCK(queue);
	taskqueue_run_locked(queue);
	TQ_UNLOCK(queue);
}

static int
task_is_running(struct taskqueue *queue, struct task *task)
{
	struct taskqueue_busy *tb;

	mtx_assert(&queue->tq_mutex, MA_OWNED);
	TAILQ_FOREACH(tb, &queue->tq_active, tb_link) {
		if (tb->tb_running == task)
			return (1);
	}
	return (0);
}

static int
taskqueue_cancel_locked(struct taskqueue *queue, struct task *task,
    u_int *pendp)
{

	if (task->ta_pending > 0)
		STAILQ_REMOVE(&queue->tq_queue, task, task, ta_link);
	if (pendp != NULL)
		*pendp = task->ta_pending;
	task->ta_pending = 0;
	return (task_is_running(queue, task) ? EBUSY : 0);
}

int
taskqueue_cancel(struct taskqueue *queue, struct task *task, u_int *pendp)
{
	u_int pending;
	int error;

	TQ_LOCK(queue);
	pending = task->ta_pending;
	error = taskqueue_cancel_locked(queue, task, pendp);
	TQ_UNLOCK(queue);

	return (error);
}

int
taskqueue_cancel_timeout(struct taskqueue *queue,
    struct timeout_task *timeout_task, u_int *pendp)
{
	u_int pending, pending1;
	int error;

	TQ_LOCK(queue);
	pending = !!callout_stop(&timeout_task->c);
	error = taskqueue_cancel_locked(queue, &timeout_task->t, &pending1);
	if ((timeout_task->f & DT_CALLOUT_ARMED) != 0) {
		timeout_task->f &= ~DT_CALLOUT_ARMED;
		queue->tq_callouts--;
	}
	TQ_UNLOCK(queue);

	if (pendp != NULL)
		*pendp = pending + pending1;
	return (error);
}

void
taskqueue_drain(struct taskqueue *queue, struct task *task)
{

	if (!queue->tq_spin)
		WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, NULL, __func__);

	TQ_LOCK(queue);
	while (task->ta_pending != 0 || task_is_running(queue, task))
		TQ_SLEEP(queue, task, &queue->tq_mutex, PWAIT, "-", 0);
	TQ_UNLOCK(queue);
}

void
taskqueue_drain_timeout(struct taskqueue *queue,
    struct timeout_task *timeout_task)
{

	callout_drain(&timeout_task->c);
	taskqueue_drain(queue, &timeout_task->t);
}

static void
taskqueue_swi_enqueue(void *context)
{
	swi_sched(taskqueue_ih, 0);
}

static void
taskqueue_swi_run(void *dummy)
{
	taskqueue_run(taskqueue_swi);
}

static void
taskqueue_swi_giant_enqueue(void *context)
{
	swi_sched(taskqueue_giant_ih, 0);
}

static void
taskqueue_swi_giant_run(void *dummy)
{
	taskqueue_run(taskqueue_swi_giant);
}

int
taskqueue_start_threads(struct taskqueue **tqp, int count, int pri,
			const char *name, ...)
{
	va_list ap;
#ifndef UINET
	struct thread *td;
#endif /* UINET */
	struct taskqueue *tq;
	int i, error;
	char ktname[MAXCOMLEN + 1];

	if (count <= 0)
		return (EINVAL);

	tq = *tqp;

	va_start(ap, name);
	vsnprintf(ktname, sizeof(ktname), name, ap);
	va_end(ap);

	tq->tq_threads = malloc(sizeof(struct thread *) * count, M_TASKQUEUE,
	    M_NOWAIT | M_ZERO);
	if (tq->tq_threads == NULL) {
		printf("%s: no memory for %s threads\n", __func__, ktname);
		return (ENOMEM);
	}

	for (i = 0; i < count; i++) {
		if (count == 1)
			error = kthread_add(taskqueue_thread_loop, tqp, NULL,
			    &tq->tq_threads[i], RFSTOPPED, 0, "%s", ktname);
		else
			error = kthread_add(taskqueue_thread_loop, tqp, NULL,
			    &tq->tq_threads[i], RFSTOPPED, 0,
			    "%s_%d", ktname, i);
		if (error) {
			/* should be ok to continue, taskqueue_free will dtrt */
			printf("%s: kthread_add(%s): error %d", __func__,
			    ktname, error);
			tq->tq_threads[i] = NULL;		/* paranoid */
		} else
			tq->tq_tcount++;
	}
#ifndef UINET
	for (i = 0; i < count; i++) {
		if (tq->tq_threads[i] == NULL)
			continue;
		td = tq->tq_threads[i];
		thread_lock(td);
		sched_prio(td, pri);
		sched_add(td, SRQ_BORING);
		thread_unlock(td);
	}
#endif /* UINET */

	return (0);
}

void
taskqueue_thread_loop(void *arg)
{
	struct taskqueue **tqp, *tq;

	tqp = arg;
	tq = *tqp;
	TQ_LOCK(tq);
	while ((tq->tq_flags & TQ_FLAGS_ACTIVE) != 0) {
		taskqueue_run_locked(tq);
		/*
		 * Because taskqueue_run() can drop tq_mutex, we need to
		 * check if the TQ_FLAGS_ACTIVE flag wasn't removed in the
		 * meantime, which means we missed a wakeup.
		 */
		if ((tq->tq_flags & TQ_FLAGS_ACTIVE) == 0)
			break;
		TQ_SLEEP(tq, tq, &tq->tq_mutex, 0, "-", 0);
	}
	taskqueue_run_locked(tq);

	/* rendezvous with thread that asked us to terminate */
	tq->tq_tcount--;
	wakeup_one(tq->tq_threads);
	TQ_UNLOCK(tq);
	kthread_exit();
}

void
taskqueue_thread_enqueue(void *context)
{
	struct taskqueue **tqp, *tq;

	tqp = context;
	tq = *tqp;

	mtx_assert(&tq->tq_mutex, MA_OWNED);
	wakeup_one(tq);
}

TASKQUEUE_DEFINE(swi, taskqueue_swi_enqueue, NULL,
		 swi_add(NULL, "task queue", taskqueue_swi_run, NULL, SWI_TQ,
		     INTR_MPSAFE, &taskqueue_ih));

TASKQUEUE_DEFINE(swi_giant, taskqueue_swi_giant_enqueue, NULL,
		 swi_add(NULL, "Giant taskq", taskqueue_swi_giant_run,
		     NULL, SWI_TQ_GIANT, 0, &taskqueue_giant_ih));

TASKQUEUE_DEFINE_THREAD(thread);

struct taskqueue *
taskqueue_create_fast(const char *name, int mflags,
		 taskqueue_enqueue_fn enqueue, void *context)
{
	return _taskqueue_create(name, mflags, enqueue, context,
			MTX_SPIN, "fast_taskqueue");
}

/* NB: for backwards compatibility */
int
taskqueue_enqueue_fast(struct taskqueue *queue, struct task *task)
{
	return taskqueue_enqueue(queue, task);
}

static void	*taskqueue_fast_ih;

static void
taskqueue_fast_enqueue(void *context)
{
	swi_sched(taskqueue_fast_ih, 0);
}

static void
taskqueue_fast_run(void *dummy)
{
	taskqueue_run(taskqueue_fast);
}

TASKQUEUE_FAST_DEFINE(fast, taskqueue_fast_enqueue, NULL,
	swi_add(NULL, "Fast task queue", taskqueue_fast_run, NULL,
	SWI_TQ_FAST, INTR_MPSAFE, &taskqueue_fast_ih));

int
taskqueue_member(struct taskqueue *queue, struct thread *td)
{
	int i, j, ret = 0;

	TQ_LOCK(queue);
	for (i = 0, j = 0; ; i++) {
		if (queue->tq_threads[i] == NULL)
			continue;
		if (queue->tq_threads[i] == td) {
			ret = 1;
			break;
		}
		if (++j >= queue->tq_tcount)
			break;
	}
	TQ_UNLOCK(queue);
	return (ret);
}

redpajama_set_name: RedPajamaGithub


Your Affordable Choice for Search Engine Optimization and Marketing | Website Design Services, Maintenance and Hosting Solutions in Comstock Park, Michigan!
Xtreme Internet Marketing™ – Creating Websites That Get You Notice! Providing Comstock Park, Michigan Clients Custom Responsive Websites, Local and National Search Engine Optimization Services.
Xtreme Internet Marketing™ works with clients from across Comstock Park, Michigan, helping with; new site designs, redesigns, updates, modifications, SEO and web hosting services. Our artistic team of dedicated professionals will bring together your vision and ideas into reality with technology, creativity, and marketing expertise. We welcome you to experience for yourself our extraordinary customer service, and creative, affordable website designs.
Comstock Park, Michigan Website Design, Search Engine Optimization and Marketing Services.
We specialize in web design and online marketing. Working with clients across Comstock Park, Michigan, big and small, creating powerful and effective websites as well as online marketing campaigns. All sites built by Xtreme Internet Marketing™ are custom, fully Google compliant, responsive, SEO-Optimized and include a FREE 1-year SSL certificate to secure customer data and increase search rankings.
Web Optimization and Analytics is a action for improving websites based on the analysis of visitors' behavior. It focuses on analyzing and optimizing marketing efforts, turning new visitors into customers and returning customers into loyal friends. Companies now have at there disposal more data about their customers than ever before as they can capture, store, analyze, and use a plethora of data from new sources to create a abundant view of the customer. The real question lies in how you use the data to spur an enviable competitive advantage.
A website is nothing if no one can find it. When potential clients search for your company you want them to find you, not your competitor. We'll help you dominate your position, and get more traffic, calls, leads and sales to grow your business. We provide local and nationwide search engine optimization – rankings, results, performance, guaranteed! We can easily say that 95% of our clients are on the First Page of Google and Yahoo Local! Can your existing SEO company say that?
Your website may be the first imprint one gets on your company and may be one of the most important pieces of advertising you have. Whether you are just starting out or are getting ready to expand into other markets, our branding identity, marketing and website design professionals, can help you reach your goals. We offer a free website evaluation, whether you need a new site built or to let you know what your existing site is doing and how we could make it better!
If you are preparing for a powerful online presence, we will put you on the map. Whether you are just starting out or are getting ready to spread out into other markets, our website design and marketing specialists, can help you reach your goals. Remember, when you hire Xtreme Internet Marketing™, we give you more than a website; we give you our loyalty, experience and professionalism. Let us help you create an eye-catching website that is custom tailored specifically to your needs. When you are ready, we're here to provide a no obligation consultation.
Our, "Listed First Local and National Organic Listings", are the Most Popular SEO Service – Offering Our Comstock Park, Michigan Clients the Best Prices in the Industry with Proven Results!
Grow Your Business Proficiently with the #1 (SEO) Search Engine Optimization Services Company in Comstock Park, Michigan.
Trust the certified experts with your online marketing plan techniques at an affordable price that gets your company found on Google, Yahoo!, Bing, and YouTube with local search marketing from Xtreme Internet Marketing™ serving Comstock Park, Michigan.
Xtreme Internet Marketing™ is Your Affordable Choice for Search Engine Optimization and Marketing Services throughout Comstock Park, Michigan!
Local and Nationwide – Results, Rankings, Performance, Guaranteed! We can comfortably say that 95 percent of our customers are on the Front Page of Google and Yahoo Local! Can your existing SEO company say that?
This is not PPC! This is a month to month service. NO CONTRACTS, you can stop anytime. We can start your paid sevice for as little as $200 per month! Call us to discover more about this amazing service – With Definite Results Guaranteed!
When potential clients search for your company you want them to discover you, not your rival. We'll make you to direct your position, and get more traffic, leads, calls and sales to grow your company.
We get your website on the Front Page of Google and Yahoo Local and Nationwide with the most affordable price in the industry – With Definite Results for those across the Comstock Park, Michigan!
Having your website highly ranked on search engines brings you more traffic from customers searching for your services or products. At Xtreme Internet Marketing™, with our (SEO) search engine optimization and Marketing solutions, it is our goal to increase your leads, calls and sales through organic search engine optimization. Our devoted team of (SEO) search engine optimization experts have the right mix of knowledge, experience and resources to achieve this goal for you, with proven results.
It's Why We've Been Named The Best Company for Search Engine Optimization (SEO) in Comstock Park, Michigan.
Our proprietary approach to the optimization procedure is highly customized and tailored to your needs, from beginning to end. There is no one solution that works for everyone. Knowing what strategies to put into use at what times, and how to put into use them for every business is precise. Beyond that, careful data collection is the key to understanding how to raise performance.
We work with local and nationwide businesses to get higher rankings online. Our teams backgrounds include specialized knowledge of: social media campaigns, marketing, branding, ecommerce, web stores, content creation, graphic design, corporate identity and logo design, API, joomla, html, wordpress, CSS, PHP, CMS, and more.
(SEO) search engine optimization is the process, of improving the amount or quality traffic to a web site from search engines via natural (organic) or un-paid search results instead of search engine marketing which deals with paid inclusion. Typically, the higher a site appears in the search results list, the more visitors it will receive from the search engine.
With that said, it refers to the procedure that make your website rank higher in organic search results, making your website more visible to people who are looking for your service, product, or brand via search engines like Google, Bing, and Yahoo.
As part of the Xtreme Internet Marketing™ (SEO) search engine optimization agreement, you submit a one-time incipient setup fee for 30 days of in-depth optimization – after which your front month starts at the monthly rate determined by our listing certified experts. This fee covers your company' category in one city. Prices for other categories and cities vary according to factors; such as number of competitors, size of city, number of locations, size of business. Volume discounts are available.
If, at the end of day 60, you are not listed on the front page, we continue optimization work for another 30 days. And if, for whatever reason, you are still not listed on page one at the end of 90 Day total, you will receive a full refund of all monthly fees – including the setup fee – NO QUESTIONS ASKED! Please read our Our Policy page for more detailed information.
This enables us to keep the costs down, passing the savings on to our clients. Our "Listed First", propitiatory software has been specifically crafted to prepare us to target local keywords across the nation in mass.
Allowing for optimization of Google's maximum one hundred and fifty characters, five keyword phrases (that a lot of keywords), to millions of undeveloped consumers within a record setting twenty four. The software propagates 28,000 plus, United States city landing pages, optimized for up to five keyword phrases, within (1) day.
Our proprietary software everything, allowing us to be the most significant local organic (SEO) search engine optimization certified experts in the world. We can now specifically target any city or state in the United States. Search engines expeditiously know and rank the listings due to our dynamically created local (SEO) search engine optimization landing pages. There is nothing that works better or faster. Nothing!
The "Listed First" software developed by Xtreme Internet Marketing™ gives us the tools that powers your website with fresh (SEO) search engine optimization content; automatically updating sites every thirty days. The tiresome updating of individual pages is now a thing in the past. We start by creating the campaign. The first 30 days of in-depth optimization. Listings normally land on the first Search Results page after the 30 days. So for the next 90 Day the organic "Listed First", landing pages will move up towards the first place of the page throughout the cities, state or nationwide based on the service purchased.
Our unique (SEO) search engine optimization local landing pages are also optimized for voice search from smart phones, these days becoming an industry standard. Local keywords are now and will be the trend for small business online marketing for many years to come. Xtreme Internet Marketing™'s "Listed First" software enables us to tackle the (SEO) search engine optimization market with targeted and incredibly fast results. It enables us to optimize your listings for 100's to 1000's of keyword phrases. We will maximize your marketing dollar by optimizing your pages for the service or delivery areas your work in or targeting, while still maintaining prime profit margins.
As part of the Xtreme Internet Marketing™ (SEO) search engine optimization agreement, you submit a one-time incipient setup fee for 30 days of intensive optimization — after which your first month begins at the monthly rate determined by our listing certified experts. This fee covers your company' category in one city. Prices for other cities and categories vary according to factors; such as number of competitors, size of city, number of locations, size of business. Volume discounts are available.
If, at the end of day 60, you are not on the local map listing, we continue optimization work for another 30 days. And if, for whatever reason, you are still not listed on the local map (within the top 7) at the end of (90) days total, you will get a full refund of all monthly fees –including the setup fee – NO QUESTIONS ASKED!
Our 90 Day, 100% money back guarantee states that if your company is not listed on the local map (within the top 7) of Google Maps by day 90, billing will be suspended until you are up, in the rare instance you are not up by day (90), you will receive a full refund of any monies paid, including the set-up fee. Our 14-day, no-pay guarantee states that if your listing drops off page one for a period of 14 consecutive days, your payments will be suspended until you return to page one and you will be issued a credit for the time lost. Please read Our Terms page for more detailed information.
Editing web content site such that it contains truly useful detailed information for site users. This involves re-appearance (naturally, not abuse) of keywords for your undeveloped customers searching Google for your services.
Instead most serious results occur in case of using harmful techniques designed to manipulate search engines: unreasonable consecutive repetition of the identical keyword (e.g.; the identical word, the identical word, the identical word) and spam (content and meta spam). Are all ways that your site can be effectively removed from the Google search rankings.
Even if a action was done flawlessly in a given time and this will bring the site in front position, nobody and nothing can guarantee that it will remain there forever. Increasing competition for first place spots periodically change, changes in the indexing algorithms are a endless threat to your position in the top rankings.
Another way to help our clients stay on the front page, not just to get there, is by registering their site in probably the most regular package of portals and web directories in the US, about 750 currently, to make certain optimization and a top rankings outside page.
SEO and Marketing Services in Comstock Park, Michigan.
Xtreme Internet Marketing™ of Comstock Park, Michigan has presented (SEO) search engine optimization for first place placement; local and national search, organic listings, PPC, social media marketing, professional website design solutions, maintenance and hosting solutions for 1000s of consumers within Comstock Park, Michigan and all across the nation! Xtreme Internet Marketing™ takes real pleasure in assisting in your businesses success. Making sure you're getting skilled solutions in (SEO) search engine optimization for first place placement in your local and national search, organic listings, PPC, Social Media Marketing (SSM), professional website design solutions, maintenance and hosting solutions! Our (SEO) search engine optimization, website designers and hosting solutions experts work their hardest to make sure your website is working for you every day!
Xtreme Internet Marketing™ is considered the signature choice for (SEO) search engine optimization for first place placement; local and national search, organic listings, PPC, social media marketing (SSM), professional website design services, maintenance and hosting solutions within Comstock Park, Michigan.
Xtreme Internet Marketing™ provides the very best (SEO) search engine optimization , maintenance and hosting solutions as soon as your website stops or your website no longer comes up on the first page when searching on Google, Yahoo!, Bing and MSN within Comstock Park, Michigan. Make our SEO, website designers and hosting solutions experts of Comstock Park, Michigan get back in business with our first class service. It is precise to us the success of your company by making sure your website, local and national search campaigns are working and back on line right away, continuing to making you money.
Xtreme Internet Marketing™ of Comstock Park, Michigan has skillful and committed (SEO) search engine optimization for first place placement; local and national search, organic listings, PPC, Social Media Marketing (SSM) experts waiting to assist you with your website design, (SEO) search engine optimization marketing services or hosting needs!
Social Media Marketing (SSM) is more important than ever. Using Xtreme Internet Marketing™ Social Media Marketing (SSM) certified pros, we can help you create a social campaign that will increase traffic, promote your brand and increase your presence within Comstock Park, Michigan and saturating the marketplace using the most widely used networking platforms such as LinkedIn, Google+, Facebook, Pinterest, and Twitter provide a great way to stay engaged with clients, build a community of users, increase brand awareness and upsell your products or services. Social Media Marketing (SSM) in many instances is a great way to offer special deals and promotions in real time to attract new customers and more traffic to your website.
Plus, conclusive content on social sites magnifies inbound links. For most social media marketing (SSM) platforms it doesn't just obtain followers; it shares your brand's message which increases visitors to your site, in turn producing new customers and referrals.
Since no two companies are alike, we afford customized social media packages for each of our clients based on their unique needs. Call us and will go over your company goals and current social media ambitions. Next, we produce a plan to get your social media work off the ground. We can also leverage your existing social media campaigns and take them to the next level.


Falcon Accounting Ltd provides cloud accounting,bookeeping, payroll, vat and annaul accounts services for businesses.
Falcon Accounting Ltd offers depth of resource and specialist skills of accountancy.
We offer cloud accounting services to property landlords.
Falcon Accounting believes in evolving with time.
As a full-service cloud accounting firm, we offer a wide range of cloud accounting services to meet the needs of a business. Since years accounting industry runs on the same rules. But in today's world, this industry applies those same rules on new technology.
The benefits of choosing cloud accounting system for business are significant and create efficiency. We make sure that clients have the right systems and processes for a faster tax return.
Falcon Accounting believes in evolving with time. Tax liabilities and traditional accounting methods evolved with evolution of digital world. Making Tax Digital (MTD) will change the core of accounting system and tax liabilities.
Choosing cloud accounting software for business provides the ability to see the financial position of client's business from anywhere, at any time, on any device. It saves time and run their business, by entering expenses on the go, or even invoicing customers from a smartphone.
At Falcon Accounting Ltd, we have a team of Chartered Accountants. They are highly experienced in all aspects of cloud accounting. They assist clients with implementing a cloud accounting system and maximising the benefits of the technology.
As a cloud accountant firm, the technology allows us to work with businesses located anywhere in the UK. Cloud-based solutions make it possible for us to provide top quality service to clients nationwide, and to our international clients as well.
Falcon Accounting Ltd offers depth of resource and specialist skills of accountancy. We help our clients expand and develop and deal with problems and opportunities as and when they arise.Our team of certified accountants ensure to provide quality accounting services in right prices.
As a full-service cloud accounting firm, we offer a wide range of cloud accounting services to meet the needs of a business. Since years accounting industry runs on the same rules. But in today's world, this industry applies those same rules on new technology. The benefits of choosing cloud accounting system for business are significant and create efficiency. We make sure that clients have the right systems and processes for a faster tax return.
Taking care of your accounts correctly is crucial and by making use of our cloud accounting services you are provided with enhanced security and ease of access. Every element of your accounts can be handled via the 'cloud', providing you with real-time viewing of your financial status at any time. By using specialist cloud accounting software, we make even the complex simple. Able to link with Xero and Cloud Accounting packages, our specialism is cloud-based online accounting.
Providing comprehensive and adept support to SMEs, Contractors, Landlords, Freelancer, YouTubers, Bloggers and Online Entrepreneurs, if you are looking for a full-service cloud accounting firm, then Falcon are the ones to call. By working in the cloud, your data is safe and secure and always accessible. You get all that you would expect from a traditional accountant, enhanced by the fact that we work with new technology, blending the two together perfectly. When you join us in the cloud you will benefit in so many ways; efficiency is increased and your accounts and tax returns dealt with swiftly and professionally. Able to view your financial position at any time, in real time, and from any device, keeping track of your accounts has never been so easy or so relaxed. Enter expenses as you go, using your smartphone and even send invoices to customers as soon as projects are completed. When working via the cloud, it does not matter where you are based in the UK as cloud-based accounting solutions can be delivered to clients not only nationwide but worldwide.
Continually keeping up with changes in technology, we are constantly evolving, upgrading our traditional accounting methods to take account of updates in the digital world. Every one of our team of accountants is a cloud expert, working alongside clients to manage their cloud accounting system and maximise the many benefits. When you choose Falcon Accounting to take care of your business, you are harnessing the power of a service that is not only unique but outstanding. Specialising in working with all types of SMEs,Contractors, Landlords, Freelancer, YouTubers, Bloggers and Online Entrepreneurs, everything that we do for you is totally accurate, transparent and reliable.
Don't make do with just any accountant when you can choose Falcon. Get in touch with us today, chat with our team and find out how we can make taking care of your accounts no longer a headache but extremely satisfying.
All businesses require accounting support and SMEs are no different. We regularly work alongside these types of companies, taking care of every aspect of accounting and dealing with HMRC as required. For a regular monthly fee we provide plentiful support so that you always know exactly how much tax you owe, paying salaries on your behalf and dealing with any director dividends.
Working with landlords throughout the UK, we ensure that all property accounts are taken care of correctly and on time and that advice is provided when required. If you are finding it hard to deal with the tax repercussions of changes to your property portfolio, we are here to assist. When we work by your side, you will be provided with your own personal accountant who will carry out precise analysis of your accounts and plan ahead to avoid any problems that may result in you having to pay extra tax.
At Falcon, we are used to dealing with contractors, providing a specialist and carefully managed service that takes care of all aspects of your accounting. In addition, we will provide expert advice supported by first-class customer service that will provide you with reliability and total peace of mind. Using our cloud-based software, we will provide a comprehensive contractor accounting service that will deal with all that you need.
Whether you are a start-up or a freelancer, at Falcon we constantly work with people within your field, providing a fixed fee accountancy service. Taking care of every business and personal taxation need, start-ups and freelancers come to Falcon for a fully assured and transparent service. Never pay more tax than you should and forget about having to deal with unexpected tax bills that send your head reeling.
Since YouTubers, bloggers and online entrepreneurs rule the digital and online world. Cloud accounting services are their companion as cloud accounting can be accessed anywhere and anytime just like blogs and videos. Being an entrepreneur can be a lonely and demanding occupation, but not when you have Falcon by your side.
Bookkeeping is a major chore, Xero makes the major and critical chore of bookkeeping enjoyable.
Which customers haven't paid their invoices?
How many bills are to be paid?
When we take care of your annual accounts we are ensuring that the financial records of your company are comprehensive and accurate. We have carefully honed and modified our service in order to take away all the hassle from you, providing both full and abbreviated statutory and annual accounts for Companies House.
Keeping track of your book-keeping can be time consuming and expensive, particularly if you are trying to do it yourself whilst running your business. By signing up to our cloud accounting service, your book-keeping will be managed and maintained by a truly professional and reliable company.
No matter what sort of business you have or what type of product or service you provide, you may need to charge and collect VAT, paying it to HM Revenue & Customs (HMRC), unless your items are VAT-free or you fall below the VAT threshold. Once you hit the magic figure and have to register for VAT, it can impact significantly upon your ability to be competitive, particularly if you are involved in production, manufacturing or retail.
Whatever type of business you operate, if you have employees it makes sense to run a formal payroll system. As a small business this can sometimes be over-looked, particularly if you have just a couple of staff and are in the habit of paying them on a cash basis, with a cheque or weekly bank transfer; the trouble with this is that you can hit problems with HMRC when you get to the end of your financial year.
As a UK-based employer then your eligible employees need to be automatically enrolled into a workplace pension. This might sound simple enough but when you are busy with other tasks, including looking after and running your business, auto enrolment can be a headache.
Falcon are your Cloud based Xero and QuickBooks Certified accountant and bookkeeper. Providing clients with access to our broad range of accountancy skills, we help your business to grow and develop, knowing that your accounts and taxes are being taken care of as they should be.
Copyright 2017 Falcon Accounting Ltd.


This week is Mental Health Awareness week and the school library provides many resource to support students mental health and wellbeing. To mark the week the Year 7 & 8 members of library café and Year 9 reading group have created positive message cards and hidden them in books around the library to spread messages of wellbeing.
One can also find recommended books on our mental health and wellbeing reading list this includes non-fiction titles to offer information and advice as well as fiction titles that explore mental health issues through the story to develop understanding and empathy.
Students can also drop into library corner on lunchtimes to speak to trained Peer Supporters for advice and information.
Download the library Mental Health & Wellbeing Reading List.