text:
\section{Introduction}
\label{sec:intro}

In this paper, we design a new neural network architecture by introducing one more dimension, called height, in addition to width and depth in the characterization of dimensions of neural networks.
We call neural network architectures with height, width, and depth as hyperparameters three-dimensional architectures.
We will show that neural networks with three-dimensional architectures improve the approximation power significantly, compared to standard networks with two-dimensional architectures (those with only width and depth as hyperparameters).
The approximation power of standard neural networks has been widely studied in recent years.
The optimality of the approximation of standard fully-connected Rectified Linear Unit (ReLU) networks (e.g., see \cite{yarotsky18a,shijun:6,shijun:2,shijun:thesis}) implies limited room for further improvements.
This
motivates us to design a new neural network architecture by
introducing an additional dimension of height beyond width and depth.

We will focus on ReLU ($\max\{x,0\}$) activation function and use it to demonstrate our ideas.
Our new network architecture is constructed recursively  via a nested structure, and hence we call a neural network with the new architecture nested network (\textbf{NestNet}).
A NestNet of height $s$ is built with each hidden neuron activated by a NestNet of height $\le s-1$. In the case of $s=1$, a NestNet degenerates to a standard network with a two-dimensional architecture.
Let us use an simple example to explain the height of a NestNet.
We say a network is activated by $(\varrho_1,{\mycdots},\varrho_r)$ if each hidden neuron of this network is activated by one of $\varrho_1,{\mycdots},\varrho_r$. Here, $\varrho_1,{\mycdots},\varrho_r$ are trainable functions mapping $\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$ to $\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$.
Then a network of height $s\ge 2$ can be regarded as
a $(\varrho_1,{\mycdots},\varrho_r)$-activated network, where $\varrho_1,{\mycdots},\varrho_r$ are (realized by) networks of height $\le s-1$.
See an example of a network of height $2$ in Figure~\ref{fig:nesnet}. The network therein can be regarded as a $(\varrho_1,\varrho_2)$-activated network, where $\varrho_1$ and $\varrho_2$ are (realized by) networks of height $1$ (i.e., standard networks). The number of parameters in the network of Figure~\ref{fig:nesnet} is the sum of the numbers of parameters in $\mathcal{L}_0,\mathcal{L}_1,\mathcal{L}_2$ and $\varrho_1,\varrho_2$.

\begin{figure}[htp!]
	\centering
	\includegraphics[width=0.64\textwidth]{NestNetEg.pdf}
	\caption{An example of a network of height $2$,   where $\varrho_1$ and $\varrho_2$ are (realized by) networks of height $1$ (i.e., standard networks).  Here, $\mathcal{L}_0$, $\mathcal{L}_1$ and $\mathcal{L}_2$ are affine linear maps.
	}
	\label{fig:nesnet}
\end{figure}

Remark that a NestNet can be regarded as a sufficiently large standard network by expanding all of its sub-network activation functions. We propose the nested network architecture since it shares the parameters in the sub-network activation functions. This is the key reason why the NestNet has much better approximation property than the standard network.
If we regard the network  in Figure~\ref{fig:nesnet} as a NestNet of height, then the number of parameters is the sum of the numbers of parameters in $\mathcal{L}_0,\mathcal{L}_1,\mathcal{L}_2$ and $\varrho_1,\varrho_2$. However, if we expand the network  in Figure~\ref{fig:nesnet} to a large standard network, then the number of parameters in $\varrho_1$ and $\varrho_2$ will be added many times for computing the total number of parameters.

Next, let us discuss our new network architecture from a perspective of hyperparameters.
We call the network architecture with only width as a hyperparameter one-dimensional architecture. Its depth and height are both equal to one. Neural networks with this type of architecture are general called shallow networks. See an example in Figure~\ref{fig:123D:net}\subref{subfig:1D}.
We call the network architectures with only width and depth as hyperparameters two-dimensional architecture. Its height is both equal to one. Networks with this type of architecture are general called deep networks. See an example in Figure~\ref{fig:123D:net}\subref{subfig:2D}.
We call the network architectures with  height, width, and depth as hyperparameters three-dimensional architecture. This is the architecture introduced in this paper.
Networks with this type of architecture are called NestNet. See an example in Figure~\ref{fig:123D:net}\subref{subfig:3D}.
One may refer to Table~\ref{tab:error:comparison} for the approximation power of networks with these three types of architectures discussed above.

\begin{table}[htp!]
	\def0.748}\begin{array}{c{1.2}
	\caption{Comparison for the approximation error of $1$-Lipshitz continuous functions on $[0,1]^d$ approximated by ReLU NestNets and standard ReLU networks.  }
	\label{tab:error:comparison}
	\vskip 0.0875in
	\centering
	\resizebox{0.999\textwidth}{!}{
		\begin{tabular}{ccccccccc}
			\toprul
			& dimension(s)  &  \#parameters &   approximation error  &  remark & reference  \\
			\midrule

			one-hidden-layer  network &
			width varies (depth=height=1) & ${\mathcal{O}}(n)$ & $n^{-1}$ for $d=1$ &  linear combination     \\

			\midrule

			deep  network &
			width and depth vary (height=1) & ${\mathcal{O}}(n)$ & $n^{-2/d}$  & composition &  \cite{shijun:2,shijun:6,shijun:thesis,yarotsky18a}\\

			\midrule

			 NestNet of height $s$ &
			width, depth, and height vary  & ${\mathcal{O}}(n)$ & $n^{-(s+1)/d}$ &  nested composition & this paper \\

			\bottomrul
		\end{tabular}

\end{table}

\begin{figure}[!htb]
	\centering
	\begin{subfigure}[c]{0.45\textwidth}
		\centering            \includegraphics[width=0.98\textwidth]{1D.pdf}
		\subcaption{}
		\label{subfig:1D}
	\end{subfigure}\hspace{12pt}

	\begin{subfigure}[c]{0.45\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{2D.pdf}
		\subcaption{}
		\label{subfig:2D}
	\end{subfigure}\\
	\vspace{6pt}\begin{subfigure}[c]{0.45\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{3D.pdf}
		\subcaption{}
		\label{subfig:3D}
	\end{subfigure}\hspace{12pt}
		\begin{subfigure}[c]{0.45\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{3D2.pdf}
		\subcaption{}
		\label{subfig:3D2}
	\end{subfigure}
	\caption{Illustrations of neural networks with one-, two-, and three-dimensional architecture.  (a) One-dimensional case (width=4, depth=height=1). (b) Two-dimensional case (width=4, depth=3, height=1). (c) Three-dimensional case (width=4, depth=3, height=3). (d) Zoom-in of an activation function of the network in (c). The network in (d) can also be regarded as a network of height $2$.}
	\label{fig:123D:net}
\end{figure}

Our main contributions can be summarized as follows.
We first propose a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth.
We show that  neural networks with three-dimensional architectures are significantly more expressive than
the ones with two-dimensional architectures.
In particular, we prove by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate Lipschitz
continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, which is much better than the optimal error $\mathcal{O}(n^{-2/d})$ achieved by standard ReLU networks with $\mathcal{O}(n)$ parameters.
In the case of $s+1\ge d$, the approximation error is bounded by ${\mathcal{O}}(n^{-(s+1)/d}) \le {\mathcal{O}}(n^{-1})$, which means we overcome the curse of dimensionality.
Furthermore, we extend our result to generic continuous functions with the approximation error characterized by the modulus of continuity. See Theorem~\ref{thm:main} and Corollary~\ref{coro:main} for more details.
Finally, we conduct a simple experiment to show the numerical advantages of the super approximation power of ReLU NestNets.

The rest of this paper is organized as follows.
In Section~\ref{sec:main},
we first present our main results  and then discuss related work.
The proofs of the main results can be found in the appendix. Next, we conduct several experiments to numerically verify our theory in Section~\ref{sec:experiments}.
Finally, Section~\ref{sec:conclusion} concludes this paper with a short discussion.

\section{Main results and related work}
\label{sec:main}

In this section, we first present our main results and then discuss related work. The proofs of the main results are placed in the appendix.

\subsection{Main results}
\label{sec:main:results}

In the rest of this paper, we will use ${\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\{n\}$ for $s\in \ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$ and $n\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi$ to denote the set of functions realized by
height-$s$ ReLU NestNets with as most $n$ parameters.
We will present the mathematical definition of ${\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\{n\}$.
We first discuss some notation regarding affine linear maps.
We use ${\mathscr{L}}$ to denote the set of all affine linear maps, i.e.,
\begin{equation*}
	{\mathscr{L}}\coloneqq
	\Big\{\mathcal{L}:\mathcal{L}({\bm{x}})={\bm{W}} {\bm{x}}+{\bm{b}},\, {\bm{W}}\in \ifmmode\mathbb{R}\else$\mathbb{R}$\fi^{d_2\times d_1},\, {\bm{b}}\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi^{d_2},\, d_1,d_2\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+\Big\}.
\end{equation*}
Let $\#\mathcal{L}$ denote the number of parameters in $\mathcal{L}\in {\mathscr{L}}$, i.e.,
\begin{equation*}
    \#\mathcal{L}=(d_1+1)d_2\quad \tn{if $\mathcal{L}({\bm{x}})= {\bm{W}} {\bm{x}}+{\bm{b}}$ for ${\bm{W}}\in \ifmmode\mathbb{R}\else$\mathbb{R}$\fi^{d_2\times d_1}$ and ${\bm{b}}\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi^{d_2}$.}
\end{equation*}
We use $\vec{g}=(\varrho_1,{\mycdots},\varrho_k)$ to denote an activation function vector, where $\varrho_i:\ifmmode\mathbb{R}\else$\mathbb{R}$\fi\to\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$ is an activation function for each $i\in \{1,{\mycdots},k\}$. When $\vec{g}=(\varrho_1,{\mycdots},\varrho_k)$ is applied to a vector input ${\bm{x}}=(x_1,{\mycdots},x_{k})$, we have
\begin{equation*}
	\vec{g}({\bm{x}})=\Big(\varrho_1(x_1),\,{\mycdots},\,\varrho_k(x_k)\Big) \quad \tn{for any ${\bm{x}}=(x_1,{\mycdots},x_k)\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi^k$.}
\end{equation*}
Let $\tn{set}(\vec{g})$ denote the function set containing all entries (functions) in $\vec{g}$. For example, if $\vec{g}=(g_1,g_2,g_3,g_2,g_1)$, then $\tn{set}(\vec{g})=\{g_1,g_2,g_3\}$.

Now we are ready to define ${\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\{n\}$ for $n,s\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi$ recursively in $s\in \ifmmode\mathbb{N}\else$\mathbb{N}$\fi$.
In the degenerate case $n=0$ or $s=0$, we define
\begin{equation*}
	{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_0\{n\}\coloneqq \Big\{\tn{id}_\ifmmode\mathbb{R}\else$\mathbb{R}$\fi,\,\tn{ReLU}\Big\}\eqqcolon {\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_s\{0\} \quad \tn{for $n,s\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi$,}
\end{equation*}
where $\tn{id}_\ifmmode\mathbb{R}\else$\mathbb{R}$\fi:\ifmmode\mathbb{R}\else$\mathbb{R}$\fi\to\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$ is the identity map.
That is, we regard the identity map and ReLU as height-$s$ ReLU NestNets with $0$ parameters or as height-$0$ ReLU NestNets with $n$ parameters.

Next, let us present the recursive step.
For $n,s\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$, a (vector-valued) function ${\bm{\phi}}\in{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\{n\}$ has the following form:
\begin{equation}\label{eq:def:bmphi}
    {\bm{\phi}}=\mathcal{L}_{m}\circ \vec{g}_m\circ
	\ {\mycdots} \
	\circ \mathcal{L}_1\circ \vec{g}_1\circ \mathcal{L}_0,
\end{equation}
where $\mathcal{L}_0,{\mycdots},\mathcal{L}_m\in {\mathscr{L}}$ are affine linear maps. Moreover, Equation~\eqref{eq:def:bmphi} shall
satisfy the following two conditions:
\begin{itemize}

    \item Condition on activation functions:
    \begin{equation}\label{eq:activaiton:functions}
        \bigcup_{i=1}^{m}\tn{set}(\vec{g}_i)=\{\varrho_1,{\mycdots},\varrho_r\}\quad \tn{ and} \quad \varrho_j\in \bigcup_{i=0}^{s-1}{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{i}\{n_j\}\quad \tn{ for $j=1,{\mycdots},r$.}
    \end{equation}
     Here, $\vec{g}_i$ is an activation function vector for each $i\in\{1,{\mycdots},m\}$. All entries in $\vec{g}_1,{\mycdots},\vec{g}_m$ form an activation function set $\{\varrho_1,{\mycdots},\varrho_r\}$, and $\varrho_j$ for each $j\in \{1,{\mycdots},r\}$ can be realized by a height-$i$ NestNet with $\le n_j$ parameters for some $i=i_j\le s-1$.
    This condition means each hidden neuron is activated by a NestNet of lower height.
    \item Condition on the number of parameters:
    \begin{equation}
    \label{eq:number:of:parameters}
        \sum_{i=0}^m \#\mathcal{L}_i \, +\, \sum_{j=1}^{r}n_j \le n.
    \end{equation}
     This condition means the total number of parameters is no more than $n$. The total number of parameters is calculated by adding two parts. The first one is the number of parameters in affine linear maps $\mathcal{L}_0,{\mycdots},\mathcal{L}_m$. The other part is the number of parameters in the activation set $\{\varrho_1,{\mycdots},\varrho_r\}$ formed by the entries in activation function vectors $\vec{g}_1,{\mycdots},\vec{g}_m$.
\end{itemize}
Then, with two conditions in Equations
\eqref{eq:activaiton:functions} and \eqref{eq:number:of:parameters}, we can define ${\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_s\{n\}$ for $n,s\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$ as follows:
\begin{equation*}
    \begin{split}
        	{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_s\{n\}\coloneqq \bigg\{
	{\bm{\phi}}: {\bm{\phi}}=\mathcal{L}_{m}\circ \vec{g}_m\circ
	\ {\mycdots} \
	\circ \mathcal{L}_1\circ \vec{g}_1\circ \mathcal{L}_0,\quad
	    \mathcal{L}_0,{\mycdots},\mathcal{L}_r\in {\mathscr{L}},\quad
		\bigcup_{i=1}^{m}\tn{set}(\vec{g}_i)
		=\{\varrho_1,{\mycdots},\varrho_r\}
		&\\
		\varrho_j\in \bigcup_{i=0}^{s-1}{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{i}\{n_j\} \  \tn{for $j=1,{\mycdots},r$}, \quad
		\sum_{i=0}^{m}\#\mathcal{L}_i \, +\, \sum_{j=1}^{r}n_j \le n &\bigg\}.
    \end{split}
\end{equation*}

Remark that, in the definition above,
$m$ can be equal to $0$. In this case, the function ${\bm{\phi}}$ degenerates to an affine linear map.

In the NestNet example in Figure~\ref{fig:nesnet}, the function $\phi$ therein is in $\bigcup_{n\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi}{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_2\{n\}$ and the activation function vectors $\vec{g}_1$ and $\vec{g}_2$ can be represented as
\begin{equation*}
	\vec{g}_1=\big(\varrho_1,\,\varrho_2,\,\varrho_1,\,\varrho_1\big)
	\quad \tn{and}\quad \vec{g}_2=\big(\varrho_2,\,\varrho_1,\,\varrho_1,\,\varrho_2,\,\varrho_2\big).
\end{equation*}
Moreover, the activation function set containing all entries in $\vec{g}_1$ and $\vec{g}_2$ is a subset of $\bigcup_{n\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi}{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{1}\{n\}$, i.e.,   $\{\varrho_1,\varrho_2\}\subseteq \bigcup_{n\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi}{\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{1}\{n\}$.

Let $C([0,1]^d)$ denote the set of continuous function on $[0,1]^d$.
By convention, the modulus of continuity of a continuous function $f\in C([0,1]^d)$ is defined as
\begin{equation*}
	\omega_f(r)\coloneqq \sup\big\{|f({\bm{x}})-f({\bm{y}})|: \|{\bm{x}}-{\bm{y}}\|_2\le r,\ {\bm{x}},{\bm{y}}\in [0,1]^d\big\}\quad \tn{for any $r\ge0$.}
\end{equation*}
Under these settings, we can find a function in ${\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\big\{{\mathcal{O}}(n)\big\}$ to approximate $f\in C([0,1]^d)$ with an approximation error ${\mathcal{O}}\big(\omega_f(n^{-(s+1)/d})\big)$, as shown in the main theorem below.
\begin{theorem}
	\label{thm:main}
	Given a continuous function $f\in C([0,1]^d)$, for any $n,s\in\ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$ and $p\in[0,\infty]$,
	there exists $\phi\in {\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\big\{C_{s,d}(n+1)  \big\}$
	such that
	\begin{equation*}
		\big\|\phi({\bm{x}})-f({\bm{x}})\big\|_{L^p([0,1]^d)}\le 7\sqrt{d}\,\omega_f\big(n^{-(s+1)/d}\big),
	\end{equation*}
	where $C_{s,d}=10^3 d^2 (s+7)^2$ if $p\in [1,\infty)$ and  $C_{s,d}=10^{d+3} d^2 (s+7)^2$ if $p=\infty$.
\end{theorem}

Remark that the constant $C_{s,d}$ in Theorem~\ref{thm:main} is valid for all $n\in \ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$. As we shall see later, $C_{s,d}$ can be greatly reduced if one only cares about large $n\in \ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$.
Generally, it is challenging to simplify the approximation error in Theorem~\ref{thm:main} to make it explicitly depend on $n$
due to the complexity of $\omega_f(\cdot)$.
However, the approximation error can be simplified to an explicit one depending on $n$ in the case of special target function spaces like H\"older continuous function space. To be exact, if $f$ is a H{\"o}lder continuous function on $[0,1]^d$ of order $\alpha\in(0,1]$ with a H\"older constant $\lambda>0$, then
\begin{equation*}\label{eqn:Holder}
	|f({\bm{x}})-f({\bm{y}})|\leq \lambda \|{\bm{x}}-{\bm{y}}\|_2^\alpha\quad \tn{for any ${\bm{x}},{\bm{y}}\in[0,1]^d$,}
\end{equation*}
implying $\omega_f(r)\le \lambda r^\alpha$ for any $r\ge 0$.
This means we can get an exponentially small approximation error $7\lambda\sqrt{d}\,n^{-(s+1)\alpha/d}$ as shown in Corollary~\ref{coro:main} below.

\begin{corollary}
	\label{coro:main}
	Suppose $f$ is a H{\"o}lder continuous function on $[0,1]^d$ of order $\alpha\in(0,1]$ with a H\"older constant $\lambda>0$. For any $n,s\in \ifmmode\mathbb{N}\else$\mathbb{N}$\fi^+$ and $p\in[0,\infty]$,
	there exists $\phi\in {\hspace{0.6pt}\mathcal{N}\hspace{-1.9pt}\mathcal{N}\hspace{-1.25pt}}_{s}\big\{C_{s,d} (n+1) \big\}$
	such that
	\begin{equation*}
		\big\|\phi({\bm{x}})-f({\bm{x}})\big\|_{L^p([0,1]^d)}\le 7\lambda\sqrt{d}\,n^{-(s+1)\alpha/d},
	\end{equation*}
	where $C_{s,d}=10^3 d^2 (s+7)^2$ if $p\in [1,\infty)$ and  $C_{s,d}=10^{d+3} d^2 (s+7)^2$ if $p=\infty$.
\end{corollary}

In Corollary~\ref{coro:main}, if $\alpha=1$, i.e., $f$ is a Lipschitz continuous function with a Lipschitz constant $\lambda>0$, then the approximation error  can be further simplified to $7\lambda\sqrt{d}\,n^{-(s+1)/d}$. See Table~\ref{tab:error:comparison} for the comparison of the approximation error of $1$-Lipshitz continuous functions on $[0,1]^d$ approximated by ReLU NestNets and standard ReLU networks.

\subsection{Related work}
\label{sec:related:work}

We will connect our results to
related existing ones for a deeper understanding. We first compare  our results to existing ones from an approximation perspective.
Next, we discuss the connection between our NestNet architecture and existing trainable activation functions.

\subsubsection*{Discussion from an approximation perspective}

The study of  the approximation power of deep neural networks has become an active topic in recent years.
This topic has been extensively studied from many perspectives, e.g., in terms of combinatorics \cite{NIPS2014_5422}, topology \cite{ 6697897},
 information theory \cite{PETERSEN2018296},
 fat-shattering dimension \cite{Kearns,Anthony:2009}, Vapnik-Chervonenkis (VC) dimension \cite{Bartlett98almostlinear,Sakurai,pmlr-v65-harvey17a},   classical approximation theory \cite{Cybenko1989ApproximationBS,HORNIK1989359,barron1993,yarotsky18a,yarotsky2017,doi:10.1137/18M118709X,ZHOU2019,10.3389/fams.2018.00014,2019arXiv190501208G,2019arXiv190207896G,suzuki2018adaptivity,Ryumei,Wenjing,Bao2019ApproximationAO,2019arXiv191210382L,MO,shijun:1,shijun:2,shijun:3,shijun:thesis,shijun:intrinsic:parameters,shijun:arbitrary:error:with:fixed:size}, etc.
 To the best of our knowledge,
 the study of neural network approximation has two main stages: shallow (one-hidden-layer) networks and deep networks.

 In the early works of neural network approximation,
 the approximation power of shallow networks is investigated.
 In particular,
 the universal approximation theorem  \cite{Cybenko1989ApproximationBS,HORNIK1991251,HORNIK1989359}, without approximation error estimate, showed that  a sufficiently large neural network can approximate a target function in a certain function space arbitrarily well.
For one-hidden-layer neural networks of width $n$ and sufficiently smooth functions, an asymptotic approximation error ${\mathcal{O}}(n^{-1/2})$ in the $L^2$-norm is proved in \cite{barron1993,barron2018approximation}, leveraging an idea that is similar to Monte Carlo sampling for high-dimensional integrals.

Recently, a large number of works  focus on the study of deep neural networks.
It is shown in \cite{shijun:2,yarotsky18a,shijun:thesis} that the  optimal approximation error is ${\mathcal{O}}(n^{-2/d})$ by using ReLU networks with $n$ parameters to approximate Lipschitz functions on $[0,1]^d$.
This optimal approximation error follows a natural question: How can we  a better approximation error?
Generally, there are two ideas to get better errors. The first one is to consider smaller function spaces, e.g., smooth functions \cite{shijun:3,yarotsky:2019:06} and band-limited functions \cite{bandlimit}. The other one is to introducing new networks, e.g.,  Floor-ReLU networks \cite{shijun:4}, Floor-Exponential-Step (FLES) networks \cite{shijun:5}, and  (Sin, ReLU, $2^x$)-activated networks \cite{jiao2021deep}.

This paper proposes a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth. 	We show that  neural networks with three-dimensional architectures are significantly more expressive than
the ones with two-dimensional architectures. To be exact,
we prove by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate Lipschitz
continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, which is better than the optimal error $\mathcal{O}(n^{-2/d})$ achieved by standard ReLU networks with $\mathcal{O}(n)$ parameters. Such a result can be generalized to generic continuous functions. As shown in Theorem~\ref{thm:main}, the approximation error is ${\mathcal{O}}\big(\omega_f(n^{-(s+1)/d})\big)$ if the target function $f$ is a continuous function on $[0,1]^d$.
 Therefore, we overcome the curse of  dimensionality if $s+1\ge d$ and  the variation of $\omega_f(r)$ as $r\to 0$ is moderate (e.g., $\omega_f(r) \lesssim r^\alpha$ for H\"older continuous functions).

\subsubsection*{Connection to trainable activation functions}

The key idea of trainable activation functions is to add a small number of trainable parameters to existing activation functions.
Let us present several existing trainable activation functions as follows.
	An ReLU-like function is introduced in \cite{7410480} by modifying the negative part of ReLU using a trainable parameter $\alpha$, i.e.,
	the Parametric ReLU (PReLU) is defined as
	\begin{equation*}
			\tn{PReLU}(x)\coloneqq
			\begin{cases}
						x & \tn{if}\  x\ge 0\\
						\alpha x &\tn{if}\  x<0.
				\end{cases}
		\end{equation*}
 A variant of
	ELU unit  is introduce in \cite{Trottier2017ParametricEL} by  adding two trainable parameters $\beta,\gamma>0$, i.e., the Parametric ELU (PELU) is given by
	\begin{equation*}
		\tn{PELU}(x)\coloneqq
		\begin{cases}
			\tfrac{\beta}{\gamma} & \tn{if}\  x\ge 0\\
				\beta\big(\exp(\tfrac{x}{\gamma})-1\big) x &\tn{if}\  x<0.
			\end{cases}
		\end{equation*}
	 Authors in \cite{8546022}  propose  a type of Flexible ReLU (FReLU), which is defined via
		\begin{equation*}
			\tn{FReLU}(x)\coloneqq
				\tn{ReLU}(x+\alpha)+\beta,
		\end{equation*}
	where $\alpha$ and $\beta$ are two trainable parameters.
One may refer to \cite{APICELLA202114} for a survey of modern trainable activation functions.
To the best of our knowledge, most existing trainable activation functions
can be regarded as \textbf{parametric} variants of the original activation functions. That is, they are attained via parameterizing the original activation functions with a small number of (typically $1$ or $2$) trainable parameters.

By contrast, activation functions in our NestNets are much more flexible. They can be (realized by) either complicated or simple sub-NestNets.
That is, we can freely determine the number of parameters in the activation functions of NestNets. In other words, in NestNets, we can randomly distribute the parameters in the affine linear maps and activation functions.
In short, compared to the networks with existing trainable activation functions, our NestNets are much flexible and have much more freedom on the choice of activation functions.

\section{Experiments}
\label{sec:experiments}

In this section
we will conduct a simple experiment to explore the numerical advantages of the super approximation power of ReLU NestNets.
To this end, we first discuss the experiment setup in Section~\ref{sec:experiment:setup} and then present the experiment results in Section~\ref{sec:experiment:results}.

\subsection{Experiment setup}
\label{sec:experiment:setup}

We will design a binary classification experiment with sufficiently many data samples. The idea comes from the Archimedean spiral, which can be described by the equation
$r=a+b\theta$
in polar coordinates $(r,\theta)$, where $a$ and $b$ are given real numbers. In other words,
we will design two sets ${\mathcal{S}}_0$ and ${\mathcal{S}}_1$ in $[0,1]^2$ based on the Archimedean spiral, where the label for ${\mathcal{S}}_i$ is $i$ for $i=0,1$.

Let us first define two curves
(Archimedean spirals)
 as follows:
\begin{equation*}
	\widetilde{{\mathcal{C}}}_i\coloneqq \Big\{(x,y):x=r_i\cos\theta,\,
	y=r_i\sin\theta,\, r_i=a_i+b_i\theta,\, \tn{for}\
	\theta\in [0,s\pi]
	\Big\},
\end{equation*}
for $i=0,1$, where $a_0=0$, $a_1=1$, $b_0=b_1=\tfrac{1}{\pi}$, and $s=20$.
To simplify the discussion below,
we normalize $\widetilde{{\mathcal{C}}}_i$ as ${\mathcal{C}}_i\subseteq [0,1]^2$, where ${\mathcal{C}}_i$ is defined by
\begin{equation*}
	{{\mathcal{C}}}_i\coloneqq \Big\{(x,y):x=\tfrac{{\widetilde{x}}}{2(s+2)}+\tfrac12,\,
	y=\tfrac{{\widetilde{y}}}{2(s+2)}+\tfrac12,\,
	({\widetilde{x}},{\widetilde{y}})\in \widetilde{{\mathcal{C}}}_i
	\Big\},
\end{equation*}
for $i=0,1$. Then, we can define the two desired sets as follows:
\begin{equation*}
	{{\mathcal{S}}}_i\coloneqq \Big\{(u,v):
	\sqrt{(u-x)^2+(v-y)^2}\le \varepsilon,\,
	(x,y)\in {{\mathcal{C}}}_i
	\Big\},
\end{equation*}
for $i=0,1$, where $\varepsilon=0.008$ in our experiments.
See an illustration for ${\mathcal{S}}_0$ and ${\mathcal{S}}_1$ in Figure~\ref{fig:spiral}.

\begin{figure}[htp!]
	\centering
	\includegraphics[width=0.42\textwidth]{spiral.pdf}
	\caption{An illustration for ${\mathcal{S}}_0$ and ${\mathcal{S}}_1$.
	}
	\label{fig:spiral}
\end{figure}
The goal of our experiments is to train a network to approximate the binary classification function $f$ well, where $f$ is defined by
\begin{equation*}
	f({\bm{x}})\coloneqq
\begin{cases}
		0 & \tn{if}\  {\bm{x}} \in {\mathcal{S}}_0 \\  1& \tn{if}\   {\bm{x}}\in {\mathcal{S}}_1.
	\end{cases}
\end{equation*}
It is easy to verify that  ${\mathcal{S}}_0$ and ${\mathcal{S}}_1$ are closed subsets of $[0,1]^2$. Then $f$ can be continuously extended to $[0,1]^2$.
Define
\begin{equation*}
	\widetilde{f}({\bm{x}})\coloneqq \frac{\tn{dist}({\bm{x}},{\mathcal{S}}_0)}{\tn{dist}({\bm{x}},{\mathcal{S}}_0)+\tn{dist}({\bm{x}},{\mathcal{S}}_1)} \quad \tn{for any ${\bm{x}}\in [0,1]^2$},
\end{equation*}
where
\begin{equation*}
	\tn{dist}({\bm{x}},{\mathcal{S}}_i)\coloneqq \inf_{{\bm{y}}\in {\mathcal{S}}_i}\|{\bm{x}}-{\bm{y}}\|_2 \quad \tn{for any ${\bm{x}}\in [0,1]^2$ and $i=0,1$.}
\end{equation*}
It is easy to verify that $\widetilde{f}$ is continuous on $[0,1]^d$ and
\begin{equation*}
    \widetilde{f}({\bm{x}})=f({\bm{x}})\quad \tn{ for any ${\bm{x}}\in {\mathcal{S}}_0\cup {\mathcal{S}}_1$.}
\end{equation*}
That means $\widetilde{f}$ is a continuous extension of $f$.
Moreover,
the modulus of continuity of the extension ${\widetilde{f}}$ can be easily estimated based on
\begin{equation*}
	\tn{dist}({\mathcal{S}}_0,{\mathcal{S}}_1)\coloneqq \inf_{{\bm{x}}\in {\mathcal{S}}_0,\,{\bm{y}}\in {\mathcal{S}}_1}\|{\bm{x}}-{\bm{y}}\|_2 >0.
\end{equation*}

Next, let us discuss the network architecture in out experiments. We consider two type of networks: NestNets and standard networks.
We adopt four-hidden-layer fully connected network architecture of width $20$, $35$, or $50$.
As for the activation function, we adopt $\sigma$ for the standard network and $\varrho$ for the NesNet, where $\sigma$ is ReLU ($\max\{x,0\}$) and $\varrho$ is realized by
a trainable one-hidden-layer ReLU network of width $3$. To be exact,
$\varrho$ is given by
\begin{equation*}
	\varrho(x)={\bm{w}}_1^T\cdot(x{\bm{w}}_0+{\bm{b}}_0)+b_1\quad \tn{for any $x\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$,}
\end{equation*}
where ${\bm{w}}_0,{\bm{w}}_1,{\bm{b}}_0\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi^3$ and $b_1\in \ifmmode\mathbb{R}\else$\mathbb{R}$\fi$ are trainable parameters. There are $10$ parameters in $\varrho$.
The initial settings for $\varrho$ in our experiments are ${\bm{w}}_0=(1,1,1)$, ${\bm{w}}_1=(1,1,-1)$, ${\bm{b}}_0=(-0.2,-0.1,0.0)$, and $b_1=0$. See Figure~\ref{fig:varrho} for illustrations.
To reduce overfitting and speed up optimization,
we take two common regularization methods:
dropout \cite{DBLP:journals/corr/abs-1207-0580,JMLR:v15:srivastava14a} and batch normalization \cite{10.5555/3045118.3045167}. The full architecture is shown in Figure~\ref{fig:arc:experiments}.

\begin{figure}[htp!]
	\centering
	\includegraphics[width=0.648\textwidth]{netarc.pdf}
	\caption{
		An illustration of the network architecture in the experiments. FC and ActFun are short of fully connected layer and activation function, respectively. ActFun=ReLU ($\sigma$) for standard networks; ActFun=sub-network ($\varrho$) for NestNets.  }
	\label{fig:arc:experiments}
\end{figure}

Finally, we are ready to present our experiment strategy. For each $i\in\{0,1\}$, we randomly choose $3\times 10^5$ training samples and $3\times 10^4$ test samples in ${\mathcal{S}}_i$ with label $i$. Then, we use these $6\times 10^5$ training samples to train our network and use these $6\times 10^4$ test samples to compute the test accuracy.
We use the cross-entropy loss function to evaluate the loss between the network and the binary classification function $f$.
The number of epochs and the batch size are set to $80$ and $64$, respectively.
We adopt Adam \cite{DBLP:journals/corr/KingmaB14} as the
optimization method.
For the $i$-th epoch, the learning rate for the parameters in $\varrho$ is $0.1\times0.003\times 0.85^{i-1}$ and the learning rate for other parameters is   $0.003\times0.85^{i-1}$. See Tables~\ref{tab:hyper:parameters} for more details.

\begin{table*}[!htb]
	\def0.748}\begin{array}{c{1.05}
	\caption{Hyperparameters in the experiments.}
	\label{tab:hyper:parameters}
	\centering
	\resizebox{0.70\textwidth}{!}{
		\begin{tabular}{cccccc}
			\toprul
			&    standard network & \multicolumn{2}{c}{NestNet} \\\midrule
			loss function  & \multicolumn{3}{c}{cross-entropy loss} \\

			\midrule
			number of training samples  & \multicolumn{3}{c}{$6\times 10^5$ in total, \
				with half in ${\mathcal{S}}_i$ for $i=0,1$} \\
			\midrule
			number of test samples  & \multicolumn{3}{c}{$6\times 10^4$  in total, \
				with half in ${\mathcal{S}}_i$ for $i=0,1$} \\

			\midrule

			number of training epochs  & \multicolumn{3}{c}{80} \\

			\midrule

			training batch size  & \multicolumn{3}{c}{256} \\

			\midrule

			optimizer  & \multicolumn{3}{c}{Adam \cite{DBLP:journals/corr/KingmaB14}} \\

			\midrule
			\multirow[c]{2}{*}[-3pt]{learning rate in $i$-th epoch} &
			\multirow[c]{2}{*}[-3pt]{$0.003\times 0.9^{i-1}$} & \footnotesize parameters in $\varrho$ & \footnotesize others\\

			\cmidrule(lr){3-4}

			&   & \footnotesize $0.1\times 0.003\times0.9^{i-1}$ &\footnotesize $0.003\times 0.9^{i-1}$\\

			\bottomrul
		\end{tabular}

\end{table*}

\subsection{Experiment results}
\label{sec:experiment:results}

We will compare the test accuracies for NestNets and standard networks. We adopt the average of largest $20$ test accuracies over all $80$ epochs as the final test accuracy.
As we can see from Table~\ref{tab:accuracy:comparison} and Figure~\ref{fig:accuracy:comparison}, via adding $10$ more parameters (stored in $\varrho$), NestNets achieve much better test accuracies   than standard networks. In an extreme comparison, the test accuracy attained by
the NestNet with $1.4\times 10^3$ parameters is still better than that of the standard network with $7.9\times 10^3$ parameters. This numerically verifies that the NestNet has much better approximation power than the standard network.

\begin{figure}[!htb]
	\centering
	\begin{subfigure}[b]{0.325\textwidth}
		\centering            \includegraphics[width=0.98\textwidth]{accuracy_epochs_w20.pdf}
		\subcaption{}
	\end{subfigure}

	\begin{subfigure}[b]{0.325\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{accuracy_epochs_w35.pdf}
		\subcaption{}
	\end{subfigure}
	\begin{subfigure}[b]{0.325\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{accuracy_epochs_w50.pdf}
		\subcaption{}
	\end{subfigure}
	\caption{Test accuracy over epochs. (a) Width=20. (b) Width=35. (c) Width=50.}
	\label{fig:accuracy:comparison}
\end{figure}

\begin{table*}[!htp]
	\def0.748}\begin{array}{c{1.05}
	\caption{Error comparison of standard networks and NestNets.}
	\label{tab:accuracy:comparison}
	\centering
	\resizebox{0.75\textwidth}{!}{
		\begin{tabular}{ccccccc}
			\toprul
			    &      width & depth & activation &  \#parameters & test accuracy  \\

			\midrule

			standard network    & \multirow{2}{*}{$20$} & \multirow{2}{*}{$4$}& ReLU ($\sigma$)  &   \multirow{2}{*}{$1.4\times 10^3$} &  0.837581 \\

			NestNet &     &    & sub-network ($\varrho$) &  & 0.981322  \\

			\midrule

			standard network  &  \multirow{2}{*}{$35$} & \multirow{2}{*}{$4$}  & ReLU ($\sigma$)  &   \multirow{2}{*}{$4.0\times 10^3$} &  0.880078 \\

			NestNet &     &    & sub-network ($\varrho$) &  & 0.999985 \\

			\midrule

			standard network  &  \multirow{2}{*}{$50$} & \multirow{2}{*}{$4$}  &  ReLU ($\sigma$)  &  \multirow{2}{*}{$7.9\times 10^3$} &  0.935018  \\

			NestNet &     &    & sub-network ($\varrho$) &  & 1.000000 \\

			\bottomrul
		\end{tabular}

\end{table*}

Next, let us discuss the trainable sub-network activation function $\varrho$.  Recall that
$\varrho$ is given by
\begin{equation*}
	\varrho(x)={\bm{w}}_1^T\cdot(x{\bm{w}}_0+{\bm{b}}_0)+b_1\quad \tn{for any $x\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi$,}
\end{equation*}
where ${\bm{w}}_0,{\bm{w}}_1,{\bm{b}}_0\in\ifmmode\mathbb{R}\else$\mathbb{R}$\fi^3$ and $b_1\in \ifmmode\mathbb{R}\else$\mathbb{R}$\fi$ are trainable parameters. There are $10$ parameters in $\varrho$.
We will compare the initial guess and the trained solution for the
parameters in $\varrho$.
Our initial settings for $\varrho$ are to make it
be a variant of ReLU except for a small neighborhood near $0$.
As we can see from Table~\ref{tab:varrho:parameters}, the parameters in $\varrho$ only change a little during the training process. However, the approximation error comparison in Table~\ref{tab:accuracy:comparison} implies that
these little changes of the parameters in $\varrho$ make a big difference.
We visualize the difference between the initial $\varrho$ and the trained $\varrho$ in Figure~\ref{fig:varrho}.

\begin{table*}[!htp]
	\def0.748}\begin{array}{c{1.05}
	\caption{Parameters in $\varrho$.}
	\label{tab:varrho:parameters}
	\centering
	\resizebox{0.985\textwidth}{!}{
		\begin{tabular}{cccccc}
			\toprul
			&  width &  ${\bm{w}}_0$ & ${\bm{b}}_0$ & ${\bm{w}}_1$ & $b_1$ \\

			\midrule

			initial  &  &  \small $(1, 1, 1)$ & \small $(-0.2,  -0.1, 0.0)$ & \small $(1, 1,  -1)$ & \small  $0$ \\
			\midrule

			\multirow{3}{*}{trained} &  20 &  \small $(0.9847, 0.9675, 1.0104)$ & \small $(-0.2223, -0.1620,  0.0607)$ & \small $(0.9902,  0.9834, -1.0138)$ & \small  $0.0033$ \\
			& 35  &  \small $(0.9809, 0.9634, 1.0140)$ & \small $(-0.2265, -0.1675,  0.0667)$ & \small $(0.9838,  0.9813, -1.0173)$ & \small  $0.0020$ \\
			& 50  &  \small $(0.9743, 0.9594, 1.0222)$ & \small $(-0.2303, -0.1634,  0.0535)$ & \small $(0.9758,  0.9726, -1.0239)$ & \small  $-0.0154$ \\

			\bottomrul
		\end{tabular}

\end{table*}

\begin{figure}[!htb]
	\centering
	\begin{subfigure}[b]{0.325\textwidth}
		\centering            \includegraphics[width=0.98\textwidth]{varrho_w20.pdf}
		\subcaption{}
	\end{subfigure}

	\begin{subfigure}[b]{0.325\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{varrho_w35.pdf}
		\subcaption{}
	\end{subfigure}
	\begin{subfigure}[b]{0.325\textwidth}
		\centering           \includegraphics[width=0.98\textwidth]{varrho_w50.pdf}
		\subcaption{}
	\end{subfigure}
	\caption{Illustrations of initial $\varrho$ and trained $\varrho$. (a) Width=20. (b) Width=35. (c) Width=50.}
	\label{fig:varrho}
\end{figure}

We use a simple example above to show the numerical advantages of the super approximation power of  NestNets, which is regarded as a proof of possibility.
It would be of great interest to further explore the numerical performance of NestNets to bridge our theoretical result to  applications. We believe that NestNets can be further developed and  applied to real-world applications.
\section{Conclusion}
\label{sec:conclusion}

This paper proposes a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth.
We show that  neural networks with three-dimensional architectures are significantly more expressive than
the ones with two-dimensional architectures.
In particular, we prove by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate Lipschitz
continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, which is much better than the optimal error $\mathcal{O}(n^{-2/d})$ achieved by standard ReLU networks with $\mathcal{O}(n)$ parameters.
Furthermore, we extend our result to generic continuous functions.
Finally, we conduct a simple experiment to show the numerical advantages of the super approximation power of ReLU NestNets.

We would like to remark that our analysis was limited on  the ReLU activation function and the (H\"older) continuous function space. It would be interesting to extend our conclusions to other activation functions (e.g., tanh and sigmoid functions) and other function spaces (e.g, Lebesgue and Sobolev spaces).
Besides, it would be interesting to  explore the numerical performance of NestNets and apply it to real-world applications.

\ifacknowledgment
\section*{Acknowledgments}
Z.~Shen is supported by Tan Chin Tuan Centennial Professorship.  H.~Yang was partially supported by the US National Science Foundation under award DMS-1945029.

\fi

{
	\small
\bibliographystyle{plain}

redpajama_set_name: RedPajamaArXiv


The terminology surrounding the different sorts of paperwork and checks which are done on a new employee can be confusing and keeps changing over time. For some jobs in Scotland, you may be asked to consent to a Basic Disclosure. The organisation which administers these checks is called Disclosure Scotland. The aim of a Basic Disclosure check is to check whether the person presenting themselves for employment has any criminal convictions.
How Long Does It Take To Receive A Standard DBS Check?
Understanding all of the terminology around police checks in the UK can be confusing, as terms keep changing, and to complicate matters further, there are separate systems running in England and Wales, Northern Ireland, and Scotland. The body which carries out checks in England and Wales is called the Disclosure and Barring Service (DBS), so the certificates they issue are normally referred to as DBS too. This body replaces the Criminal Records Bureau checks, which were known as CRB. There are many different reasons why someone may apply for a standard DBS check, and the length of time to receive the certificate will depend on the individual's circumstances.
What is Checked in a CRB Check?
It's a commonly seen phrase when you're browsing through job adverts or even on boards advertising for volunteers – "subject to satisfactory references and CRB or DBS check". Most people are clear about what the references mean, but there is a lot more confusion about the other part of the process. CRB, or Criminal Records Bureau, is actually an obsolete term. In 2012, CRB merged with another body called the Independent Safeguarding Authority, and a new body called Disclosure and Barring Service (DBS) was set up. The functions of this new body are broadly similar, and in the future this is the organisation which will carry out whatever checks may be required.
The most detailed type of Police check carried out in Scotland is the Enhanced Disclosure. Until very recently, this was the type of check carried out on people working or volunteering with children or other vulnerable people, until that sort of checking was replaced by the Protecting Vulnerable Groups (PVG) scheme. Enhanced Disclosure still exists though, and there are a number of situations in which you might have to apply for one.
What Is A Basic Disclosure Form?
A Basic Disclosure form is the simplest form of police check available from Disclosure Scotland, for those people intending to apply for certain jobs in Scotland, or applying to the Home Office for an immigration visa and who are living at an address in Scotland. The form is the starting point in the process of having your police record officially checked, and to make things easier, this type of form can be completed and submitted online.
How Long Does It Take for a DBS Check to Come Back?
There are lots of different occupations which require the employee to undergo a DBS check. The problem with the process is that it can be quite lengthy, and involves the employee filling in a form listing their personal details, the employer verifying their personal identification documents, then the form being sent off to the DBS for the police checks to be done. On average, this process can take 8 weeks. If someone has changed their name a lot and has moved house dozens of times, the checking may take even longer. There is no way around this process and no way of speeding things up – there is no option to pay extra for an "express service". Bear in mind that it may take several weeks for a DBS check to be returned when you apply for a new position, and always complete forms as a matter of urgency to keep the process moving as smoothly as possible. There are however some options for both employer and employee while waiting for a DBS check to be completed.
When Do I Need To Use a Disclosure Certificate?
There are lots of different circumstances under which you might be asked to produce a Disclosure certificate. Disclosure is the name given to the process of having your criminal record checked before applying for a job or voluntary position in Scotland. In England, the very similar process is called a DBS check. There are many, many reasons why someone may apply for one of the different types of Disclosure certificate, but here are the most common.
There are different levels of police checking certificates which are available to employers throughout the UK, and the more detailed type of check in England and Wales is called an Enhanced DBS check. This sort of check is carried out on both employees and volunteers who work with vulnerable adults, people who work in schools or childcare establishments, foster carers, healthcare professionals, carers and social workers. There are two types of enhanced checks, one which gives simply the police records of the person concerned, and one which also searches the barred lists of people working in certain occupations who have been barred from working with children or vulnerable adults. The type of check carried out will depend on the exact position being applied for, and in the majority of cases, the employer will know what type of check is needed.


This is the easiest workshop you will ever take! If you are interested in the fiber arts but are too afraid of knitting or sewing...this is a great first class for you!
In this class you will learn the basic properties of felting with wool. Using your new found skill, you will create 2 marbelized felted soaps. We will supply gift tags in case you are willing to part with your creations...they make excellent gifts!
This event is BYO food + drink. Cheers!
You must purchase a ticket to reserve your seat! We require 5 tickets sold to avoid cancellation of the workshop. In the event of cancellation, you will receive a full refund.


text:
Do you think we'll see Toy Story 4 sometime soon?
Toy Story is one of our all-time favourite films at Fun Kids!
It's funny, silly and looks out-of-this-world.
So we were gobsmacked when we heard there might be a 4th film on the way!
More to click...
Emma Watson cast as Belle in new Beauty and the Beast film!
The Hobbit actor John Bell goes back to school with Into Film!
Into Film chat to the cast of The Fault In Our Stars!
A very important guy in the film industry tweeted that 'Toy Story 4 will be out in 2015'. Pretty exciting right?
Well, maybe. The people who make the film – Pixar – then said that 'nothing official' has been announced. That definitely sounds like it could still be happening unofficially though, which would be amazing.
Even though they haven't confirmed Toy Story 4, Pixar have announced 'Finding Nemo 2' will be out in 2016 which is great news as the original film was awesome.
If you like hearing the latest movie gossip then tune in to Hannah's Weekend every Saturday from 10am! A film expert from FILMCLUB comes in to tell us about the latest film news.
Tune in to Fun Kids on DAB Digital Radio in London and online across the UK! Click here to listen!
And here's the latest look at the new Pixar movie. It's 'Monsters University'…
Playing: shaun the sheep (farmageddon remix) by nadia rose
Into Film gets the scoop on Interstellar!
This week's best movies for kids!
Watch Into Film chat to funny man Steve Carell!
yaya! love toy storyy. Buzz is my favourite becaus hes got a silly chin.
hey I think this is ACE we r baking your rice crispie cakes Yay!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1

redpajama_set_name: RedPajamaCommonCrawl


text:
Eluana Englaro Death Sparks Outrage
Eluana Englaro, 38, died last night at a clinic in Udine in Northeastern Italy, where she had been for the past week, as the previous posts have tried to keep on top of the developments of this case, hoping and praying for a miracle. This debate has rocked Italy and caused huge debates between pro-life and pro-death activists.
Her father briefly spoke out and confirmed that she had died and said he had nothing to say but that he would prefer to be alone at this time.
After a battle to let his daughter die by removing her survival needs of food and water for the last 10 years claiming it was the wishes of his daughter Eluana Englaro, her father, Beppino Englaro, won his case within Italy's top court authorizing in November 2008 his right-to-die case that has Italy in a total uproar along with opinions from all over the world.
Eluana Englaro, died 4 days after her feeding tubes and hydration apparatus was slowly removed in a private hospital. This outrage and inhuman case will absolutely most definitely always stay in the minds of many and will likely be used as a benchmark to finalize changes to laws regarding euthanasia. The Englaro case has been compared to that of American Terri Schiavo, who spent 15 years in a vegetative state and was allowed to die in 2005 after a long court battle.
While in the midst of debating a bill and a vote that would have saved her life and her feeding tube re-instated Englaro died.
Although no cause of death has been announced news reports have indicated because of such a quick death were doses of sedatives given to her? Palliative medication in high does causes premature death.
The moving of Englaro's to a private hospital and her dehydration death was met with angry protesters throughout Italy during the weekend and over the past few weeks. She became part of all pro-life catholic supporters and everyone's hopes and prayers.
Emergency legislation through the Italian Parliament to save Englaro's life after an earlier attempt was blocked by the country's president, a former communist. However, the legislation was still in process when Englaro's death was announced sadly.
It seems now there are a lot of questions as to the surprise quickness of her death according to the Euthanasia Prevention Coalition International.
"To intensely dehydrate a person to death dehumanizes them because it denies them the basic care due to a human person. We turn them into an object," he told LifeSiteNews. "Everybody deserves basic care, which includes food, fluid, and warmth as long as it is necessary to sustain life. This is not extraordinary treatment."
"We ask the question, how did she actually die? She wouldn't have died in just a few days of dehydration," he added. What was the cause of death?
Despite court rulings in favor of removing her tubes, Italy's government issued an emergency decree Friday to help her or anyone in a vegetative state from having tubes disconnected.
They had the decree issued despite court rulings and a warning from Italy's president that the government shouldn't intervene in the case. This decree passed at Cabinet on Friday stating feeding and giving water to the incapacitated or those who depend on it cannot be taken away.
The issue of this decree was against the advice of Giorgio Napolitano, who as Italy's president is the country's top official. His argument was that the decree would violate a final ruling by a court.
Since Mr. Napolitano has to sign any government decree, the Cabinet decision to move ahead despite the president's recommendation might lead to a bitter battle between two of the country's most senior officials.
Italian Prime Minister, Berlusconi, tried desperately to expedite the decree claiming it was urgent and crucial for her survival and to disconnect Ms. Englaro's feeding tube which the process had begun last Friday wasnothing less than slow murder .... he was unsuccessful time wise in saving this particular woman, god bless her and everyone who tried to save her from the tragic life she led here.
(SAST/CAT: GMT+2), this article was posted by: A friend of a Saint On Wednesday, February 11, 2009
Labels: A Friend Of A Saint Posts
Media Study South Africa: Danny Boyle ("Slumdog Mi...
Vatican, Europe; Great Britain; Europe: SSPXer's a...
Brazil: priest suspended for heresy, condoms, a de...
South Africa: "Hurley's War": Former Durban Bishop...
Britain / Argentina; SSPX schism: Williamson Retur...
Ash Wednesday: a point of what?
Atheists Bus Ads & Ash Wednesday
Cdn Military: Immediate Investigation
Al-Qaeda holds two Canadians Prisoner
Sundin Back last night playing Against TO
Obama and Embryonic Stem Cells Research
Holocaust Bishop ... 10 days to Leave Argentia
Phillipine Troops arrest Two Foreign Terrorists
Kidnapped Italian Nuns Released in Kenya
Men and Women ... We Sin Differently
Bishop, Episcopal Church, Thrown in Jail
Asking for Forgiveness
Roman Catholic Celibate Priests
Washington: USA, North America: watching the desse...
The loss of true seeking of knowledge at many Univ...
FSSP launches Prayer for Benedict XVI: the FSSP le...
Journey: Projected illusion: waves of light, of so...
Southern African Catholic Bishops Conference: "Jul...
South Africa: Southern African Catholic Bishops' C...
Sri Lanka; Asia; Eurasia: Tamil Tigers shoot nuns,...
Mexican Bishops Distrustful of politicians
EU, Europe: Caritas Spain, Germany: to help Street...
Vatican to EU: stop ignoring African human Rights
Former Communist Block Bishops: called to co-opera...
Priests in South Africa in schism?
Communications with Pope Benedict XVI
SSPX sidelines Holocaust Deniers
Social Justice: Central Africa: Bishops blame gove...
Italy appoints first female judge of Canon Law
A Very Cool Media Commercial or Advertisement !!
Turmoil continues as Protesters Injured in Madaga...
Madagascar : Mayor continues campagn: government's...
Vatican Library Renovations: Video from Vatican on...
Benedict XVI: say no to Euthanasia
Agenzia Fides News: 07 / Feb / 2009
Journey:Twilight essence, Blue ribbon of Destiny: ...
Novartis nearly sneak into Vatican Radio Interview...
Latin mass Schedule in Johannesburg for the Coming...
Italy, EU, Europe: Catholic Prolife objects: Fathe...
Venezuela; Chavez supporters attack Jews; Vatican ...
BBC Reports on the Papacy's strongest yet condemna...
Father Proceeds to "Murder" his Daughter in 3 days...
Reunification of Russia's East and Vatican's West:...
The drowning, swirling Catholic Church, the electr...
Canada: Farewell to 108th Cdn Soldier killed in A...
Southern African Catholic Bishops Conference Answe...
February 2, Canada, did the Groundhog see his shad...
A look at a family after a war in Gaza...
Great Britain Planning, inciting Mass genocide?
(Traditionalism) TAC Anglicans to join Catholic ch...
Zimbabwe, SADC Region: Why would Morgan Tsvangirai...
I trust God


text:
Bombardier's Thought-provoking Safety Standdown 2019 Shines the ...
Bombardier Announces Definitive Agreement to Sell Aerostructures Business to Spirit AeroSystems Holding, Inc.
Sale of aerostructures business supports Aviation transformation, refocuses on business aviation and strengthens liquidity
Expected cash proceeds of $500 million (1) plus the assumptions of liabilities
Transaction implies enterprise value to 2019E EBITDA multiple of approximately 10x
All amounts in this press release are in U.S. dollars unless otherwise indicated.
Bombardier (TSX: BBD.B) today announced a definitive agreement to sell its aerostructures business to Spirit AeroSystems Holding, Inc. (Spirit), supporting Bombardier's strategic decision to focus on its two strong growth pillars, trains and business aircraft.
With this transaction, Spirit will acquire Bombardier's aerostructures activities and aftermarket services operations in Belfast, U.K.; Casablanca, Morocco; and its aerostructures maintenance, repair and overhaul (MRO) facility in Dallas, U.S. for a cash consideration of $500 million and the assumption of liabilities with a total carrying value in excess of $700(2) million, including government refundable advances and pension obligations. Following the transaction, Spirit will continue to supply structural aircraft components and spare parts to support the production and in-service fleet of Bombardier Aviation's Learjet, Challenger and Global families of aircraft.
2019 revenues for these activities are expected to be approximately $1.0 billion(3) , while generating adjusted EBITDA margin of approximately 12%. On this basis, the transaction implies an enterprise value to EBITDA multiple of approximately 10x.
The transaction follows the formation of Bombardier Aviation earlier this year and streamlines Bombardier's aerostructures footprint to focus on core capabilities in Montreal, Mexico and its Global 7500 wing operations in Texas. The transaction also further strengthens Bombardier's liquidity as it moves toward the deleveraging phase of the turnaround. The transaction is expected to close in the first half of 2020 and remains subject to regulatory approvals and customary closing conditions.
"This transaction represents another strategic milestone in the reshaping of our portfolio to focus on our strong business aircraft and rail franchises," said Alain Bellemare, President and Chief Executive Officer, Bombardier Inc. "We are confident that Spirit's acquisition of these aerostructures assets is the best outcome for customers, employees and shareholders, and we are committed to ensuring a smooth and orderly transition."
Bombardier, Challenger, Global, Global 7500, and Learjet are trademarks of Bombardier Inc. or its subsidiaries.
Represents the carrying values of liabilities for Bombardier as of September 30, 2019. This value may differ under Spirit ownership.
Including approximately one third from internal sales to Bombardier Aviation.
Home > Media > News > Bombardier Announces Definitive Agreement to Sell Aerostructures Business to Spirit AeroSystems Holding, Inc.


Beautiful, Elegant. These Sankofa birds perfectly frame a beautiful trinity setting, emphasizing the wearer's and relationship's past, present & future. The present stone is largest, of course as a reminder that all power lay in the present moment. The Sankofa bird has its feet planted forward, arching its neck to retrieve an egg from its back. Literally meaning to 'go back and fetch it', this symbol represents introspection, honoring and learning lessons from one's past and building a future benefited by hindsight.
SYMBOL: Ghana: Adinkra: Sankofa. Please see our symbols page for information. Sizes 4.5- 8.5.
DIMENSIONS: Band Width Tapers: 1mm-2mm. Sankofa Bird is approx 3mm.
Stunning Emerald Cut Diamond framed in paved accent diamonds, and our signature 18k White Gold semi mounting with Gye Nyame Undercarriage. Semi mount: 18k White Gold.
Center Stone: 1.20 ct, Emerald Cut. Color: G-H. Clarity: VS2.
Accent Diamonds: .50 tcw, paved. Color: G-H. Clarity: VS2.
Available in Platinum, 14K Yellow & Rose. Please call for an appointment or more information.
OPTIONS: Available in platinum, 14k& 18k white, rose or yellow. Matching earrings available Please call for pricing. Orders take 2-4 business days if not in stock. Rush orders available. Please call for more information.
Clean, sharp, distinct. 14k White Gold band with 6 symbols and 6 diamonds in between each symbol. SYMBOL: Ghana: Adinkra: Gye Nyame. Price: $1480 sizes 5-6. Sizes 6 1/4-9 $1773. Sizes 9-12 $1983.
Diamonds: Color G-H, Clarity VS. Shape: Princess. Diamond total weight .18pts (.3 pts each).
Available in 8mm and 10mm widths. Other symbols, round diamonds and other gemstones available. Custom orders take 4-6 weeks. Rush orders available. Please call for more information.
Smart & Elegant. Brushed White 14k Gold Band. Symbol: Gye Nyame. Design:3 symbols with 4 diamonds between each symbol. 8 diamonds total.Design extends to the sides of the ring.
Diamonds: Color G-H, Clarity VS. Shape: Princess. Diamond total weight .24pts (.3 pts each).
Available in 8mm and 10mm widths. 18k & Platinum, other symbols, round diamonds and other gemstones available. Price $2425 for sizes 5-9, Sizes 9 1/4-12 $2625, Sizes 12 1/4-16 $2825.
14k White Gold. Mounting diamonds total weight .25 carats. Color H-I Clarity VS-SI. Set your own center stone, or call us for a quote on loose diamonds. Prongs for center stone can be custom sized. Engraving available. Adinkra symbol: Gye Nyame. Symbol explanation on our Symbols page. Custom orders take 4-6 weeks. Rush orders available. Please call for more information.
14k White Gold. Side diamonds total weight .25 carats. Color H-I Clarity VS-SI. Set your own center stone, or call us for a quote on loose diamonds. Prongs for center stone can be custom sized. Engraving available. For sizes 4-9. For larger sizes, please call. Adinkra symbol: Gye Nyame. Symbol explanation on our Symbols page. Custom orders take 4-6 weeks. Rush orders available. Please call for more information.


There are principled and pragmatic reasons for the Chancellor to act in his first Budget of the new parliament.
George Osborne leaves Downing Street for Prime Minister's Questions on June 3, 2015. Photograph: Getty Images.
Throughout the election campaign, the Tories refused to rule out again reducing the top rate of tax, from 45p to 40p. David Cameron and George Osborne would not have maintained this ambiguity unless they sincerely wanted to keep this option open. It is unsurprising, then, that ahead of his emergency Budget on 8 July, the Chancellor is contemplating just this move (as his shadow Chris Leslie recently warned he would).
Osborne wanted to reduce the rate from 50p to 40p in his 2012 statement but was prevented from doing so by the Lib Dems (who demanded a mansion tax in return). With the Conservatives now alone in government for the first time in 18 years, this political obstacle has been removed.
After the slump in the Tories' popularity that followed the first rate cut, plenty will ask how Osborne could possibly be tempted to take the same path again. But there are both principled and pragmatic reasons for him to do so. The first and most salient is that the Chancellor does not believe in the 45p rate. He regards it as an economic hindrance that is outmoded in a globalised world (just four EU member states have a higher rate). Osborne is typically viewed as a political pragmatist par excellence. But his decision to cut the rate from 50p to 45p was an act of pure conviction - he was prepared to accept the firestorm that followed.
Osborne wants to leave the Treasury (potentially for No.10) with the top rate returned to 40p. He will not have a better opportunity to act than his first Budget. "If it were done when 'tis done, then 'twere well it were done quickly." The Tories are in their honeymoon period (when voters give them the benefit of the doubt) and Labour and the Lib Dems are leaderless. When the opposition protest, Osborne will be able to point out that he has merely returned the rate to its level for 155 of the 156 months that Labour was in power. The 2012 "omnishambles" did not stop the Conservatives winning a majority in May. Any damage from cutting the top rate now will likely have dissipated by 2020.
The Tories may yet baulk at the prospect of cutting taxes for the top 1 per cent of earners while cutting tax credits for those at the bottom. Osborne would be wise to balance the move with tax rises on the wealthy elsewhere in the Budget. But do not be surprised if Osborne resolves that the opportunity is too good to waste.
› When it comes to advertising food, why do lesbians get yoghurt?


text:
Micrometopus punctipennis is een keversoort uit de familie boktorren (Cerambycidae). De wetenschappelijke naam van de soort is voor het eerst geldig gepubliceerd in 1885 door Quedenfeldt.

Boktorren

redpajama_set_name: RedPajamaWikipedia


text:
Donauprinzessin steht für:

 Donauprinzessin (Fernsehserie), deutsche Fernsehserie
 Rossini (Schiff), ein Schiff das bis 2006 Donauprinzessin hieß
 Donauprinzessin (Schiff, 1991), ein Schiff das bis 2021 Stadt Bad Ems hieß