text:
\section{Introduction}

Statistical learning theory provides  tight and illuminating results under its assumptions and for its objectives (e.g., \citealt{vapnik1998statistical,mukherjee2006learning,mohri2012foundations}). As the training datasets are considered as random variables, statistical learning theory was initially more concerned with the study of  \textit{data-independent} bounds based on the capacity of the hypothesis space \citep{vapnik1998statistical}, or the classical  stability of learning algorithm \citep{bousquet2002stability}. Given the observations that these data-independent bounds could be overly pessimistic for a ``good'' \textit{training dataset}, \textit{data-dependent} bounds have also been  developed in statistical learning theory, such as  the \textit{luckiness framework} \citep{shawe1998structural,herbrich2002algorithmic}, \textit{empirical} Rademacher complexity of a hypothesis space \citep{koltchinskii2000rademacher,bartlett2002model}, and the robustness of learning algorithm \citep{xu2012robustness}.

Along this line of reasoning, we notice that the previous bounds, including data dependent ones, can be   pessimistic for a ``good'' \textit{problem instance}, which is defined by a tuple of a true (unknown) measure, a training dataset and a learned model (see Section \ref{sec:basis} for further details). Accordingly, this paper  proposes a learning theory designed to be strongly dependent on each individual problem instance. To achieve this goal, we directly analyse the generalization gap (difference between expected error and training error) and datasets as  non-statistical  objects via measure theory. This is in contrast to the setting of statistical learning theory wherein these objects  are treated as random variables.

The non-statistical nature of our proposed theory can be   of practical interest on its own merits. For example, the non-statistical nature  captures  well a  situation wherein a training dataset is specified and fixed first  (e.g., a UCL dataset, ImageNet, a medical image dataset, etc.), rather than remaining random with a certain  distribution.
Once a dataset is actually specified, there is no randomness remaining over the dataset (although one can artificially create randomness via an empirical distribution). For example, \citet{zhang2016understanding} empirically observed that \textit{given a fixed (deterministic) dataset} (i.e., each of CIFAR10, ImageNet, and MNIST), test errors  can be  small despite  the large capacity of the hypothesis space and possible instability of the learning algorithm. Understanding and explaining this empirical observation has  become an active research area  \citep{arpit2017closer,krueger2017deep,hoffer2017train,wu2017towards,dziugaite2017computing,dinh2017sharp,bartlett2017spectrally,brutzkus2017sgd}.

For convenience within this paper, the proposed theory is called   \textit{analytical learning  theory}, due to its non-statistical and analytical nature. While the scope of  statistical learning theory covers both prior and posterior guarantees, analytical learning theory focuses on providing \textit{prior} insights  via \textit{posterior} guarantees; i.e.,
the mathematical bounds are available before the learning is done, which provides insights a priori to understand the phenomenon and to design algorithms, but the numerical value of the bounds depend on the posterior quantities. A firm understanding of analytical learning theory requires a different style of thinking and a shift of technical basis from statistics (e.g., concentration inequalities) to measure theory.  We present the foundation of analytical learning theory in Section \ref{sec:basis} and  several applications in Sections \ref{sec:classical_ML}-\ref{sec:method}.

\section{Preliminaries}

 In machine learning, a typical goal is to return  a model  $\hat y_{\mathcal A(S_{m})}$ via a learning algorithm  $\mathcal A$ given a dataset $S_{m}=\{s^{(1)},\dots,s^{(m)}\}$ such that  the expected error   $\EE_{\mu}[L \hat y_{\mathcal A(S_{m})}]\triangleq \EE_{z}[L \hat y_{\mathcal A(S_{m})}(z)]$ with respect to  a true (unknown)  normalized measure $\mu$  is minimized. Here,    $L\hat y$ is a function that combines a loss function $\ell$ and a  model $\hat y$; e.g.,
in supervised learning,  $L\hat y(z)=\ell(\hat y(x),y)$, where $z=(x,y)$ is a pair of an input $x$ and a target $y$.
 Because the expected error $\EE_\mu[L  \hat y_{\mathcal A(S_{m})} ]$ is often not computable, we usually approximate the expected error by an empirical error  $\hat \EE_{Z_{m'}}[L \hat y_{\mathcal A(S_{m})}]\triangleq\frac{1}{m'}\sum_{i=1}^{m'} L \hat y_{\mathcal A(S_{m})}(z^{(i)})$ with a dataset $Z_{m'}=\{z^{(1)},\dots,z^{(m')}\}$. Accordingly, we define
$\text{\textit{the generalization gap}} \triangleq \EE_\mu[L  \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$.  One of  the goals of learning theory is to explain and validate  when and how minimizing $\hat \EE_{S_{m}}[L \hat y_{\mathcal A(S_{m})}]$ is a sensible approach to minimizing $\EE_\mu[L  \hat y_{\mathcal A(S_{m})}]$ by analyzing the generalization gap, and to provide bounds on the performance of $\hat y_{\mathcal A(S_{m})}$ on new data.

\subsection{Discrepancy and variation}

In the following, we define a quality of a dataset, called \textit{discrepancy}, and a quality of a function, called \textit{variation in the sense of Hardy and Krause}. These definitions have been used in harmonic analysis, number theory, and numerical analysis  \citep{krause1903fouriersche,hardy1906double,hlawka1961funktionen,niederreiter1978quasi,aistleitner2017functions}. This study adopts these definitions  in the context of machine learning. Intuitively, the \textit{star-discrepancy} $D^{*}[T_{m},\nu]$ evaluates how well a  dataset $T_m=\{t^{(1)},\dots,t^{(m)}\}$ captures a normalized  measure $\nu$, and the \textit{variation $V[f]$ in the sense of Hardy and Krause}   computes how a function $f$ varies in total w.r.t.  each small perturbation of every cross combination of its variables.

\subsubsection{Discrepancy of dataset with respect to a measure}

 For any $t=(t_{1},\dots,t_{d})\in[0,1]^{d}$, let $B_t \triangleq [0,t_1] \times  \cdots  \times [0,t_d]$ be a closed axis-parallel box with one vertex at the origin. The \textit{local discrepancy} $D[B_t;T_{m},\nu]$ of a dataset $T_m=\{t^{(1)},\dots,t^{(m)}\}$ with respect to a normalized Borel measure $\nu$ on a set $B_t$ is defined as
\begin{displaymath}
D[B_t;T_m,\nu] \triangleq \left(\frac{1}{m}\sum_{i=1}^m \one_{B_{t}}(t^{(i)}) \right) -\nu \big(B_{t} \big)
\end{displaymath}
where $\one_{B_{t}}$ is the indicator function of a set $B_{t}$. Figure \ref{fig:local_discrepancy} in Appendix \ref{app:illustration_discrepancy} shows an illustration of the  \textit{local discrepancy} $D[B_t;T_{m},\nu]$ and related notation.
The \textit{star-discrepancy} $D^{*}[T_{m},\nu]$ of a dataset $T_m=\{t^{(1)},\dots,t^{(m)}\}$ with respect to a normalized Borel measure $\nu$ is defined as
\begin{displaymath}
D^*[T_{m},\nu] \triangleq \sup_{t\in[0,1]^{d}} \big| D[B_t;T_m,\nu] \big|.
\end{displaymath}

\subsubsection{Variations of a function}
 Let $\partial_l$ be the partial derivative operator; that is, $
\partial_l g(t_{1},\dots,t_{k})
$
is the partial derivative of a function $g$ with respect to the $l$-th coordinate at a point $(t_{1},\dots,t_{k})$. Let  $\partial_{1,\dots,k}^k \triangleq\partial_{1},\dots,\partial_{k}$. A partition $P$ of $[0,1]^k$  with size $m_1^{P},\dots, m_k^P$ is a set of finite sequences $t^{(0)}_l,t^{(1)}_l\dots,t^{( m_l^P)}_l$ ($l=1,\dots,k$) such that $0=t^{(0)}_l \le t^{(1)}_l \le \cdots \le t^{(m_l^{P})}_l=1$ for $l=1,\dots,k$. We define a difference operator $\Delta^P_{l}$   with respect to a partition $P$ as: given a function $g$ and a point $(t_{1}, \dots t_{l-1},t_{l}^{(i)},t_{l+1},\dots, t_{k})$ in the partition $P$ (for $i=0,\dots,m_l^P-1$),
\small
\begin{align*}
\Delta^P_{l} g(t_{1}, \dots t_{l-1},t_{l}^{(i)},t_{l+1},\dots, t_{k})
 = g(t_{1}, \dots t_{l-1},t_{l}^{(i+1)},t_{l+1},\dots, t_{k})
 - g(t_{1}, \dots t_{l-1},t_{l}^{(i)},t_{l+1},\dots, t_{k}),
\end{align*}
\normalsize
where $(t_{1}, \dots t_{l-1},t_{l}^{(i+1)},t_{l+1},\dots, t_{k})$ is the subsequent  point in the partition $P$   along the coordinate $l$. Let  $\Delta^P_{1,\dots,k}\triangleq\Delta^{P}_{1} \dots  \Delta^{P}_{k}$.
Given a  function $f$ of $d$ variables, let $f_{j_{1}\dots j_k}$ be the  function restricted on $k\le d$ variables such that $f_{j_{1}\dots j_k}(t_{j_{1}},\dots,t_{j_k}) = f(t_{1},\dots,t_d)$, where  $t_{l} \equiv 1$ for all $l \notin \{j_{1},j_2,\dots j_k\}$. That is, $f_{j_{1}\dots j_k}$ is a function of $(t_{j_1}, \dots,t_{j_k})$ with other original variables being fixed to be one.

The \textit{variation of $f_{j_{1}\dots j_k}$ on $[0,1]^k$ in the sense of Vitali} is defined as
\begin{align*}
V^{(k)}[f_{j_{1}\dots j_k}]
\triangleq\ \sup_{P \in \mathcal{P}_k} \sum_{i_1=1}^{m^P_1-1} \dots \sum_{i_{k}=1}^{m^P_k-1} \left|\Delta^P_{1,\dots,k} f_{j_{1}\dots j_k}(t_{j_1}^{(i_{1})}, \dots,t_{j_k}^{(i_{k})}) \right|,
\end{align*}
where $\mathcal{P}_k$ is the set of all partitions of $[0,1]^k$. The \textit{variation of $f$ on $[0,1]^d$ in the sense of Hardy and Krause} is defined as
\begin{displaymath}
V[f] = \sum_{k=1}^d  \ \ \sum_{1\le j_1 <  \dots < j_k \le d} V^{(k)}[f_{j_{1}\dots j_k}].
\end{displaymath}

For example, if $f$ is linear on its domain, $V[f] = \sum_{1\le j_1 \le d} V^{(1)}[f_{j_{1}}]$ because $V^{(k)}[f_{j_{1}\dots j_k}]=0$ for all $k>1$. The following proposition might be helpful in intuitively understanding the concept of the variation as well as   in computing it when applicable.  All the proofs in this paper are presented in Appendix \ref{app:proofs}.

\begin{proposition} \label{prop:variation_diff}
Suppose that $f_{j_{1}\dots j_k}$ is a function for which $\partial_{1,\dots,k}^k f_{j_{1}\dots j_k}$ exists on $[0,1]^k$. Then,
\small
$$
V^{(k)}[f_{j_{1}\dots j_k}] \le \sup_{(t_{j_{1}},\dots,t_{j_k})\in[0,1]^{k}} \left|\partial_{1,\dots,k}^k f_{j_{1}\dots j_k}(t_{j_{1}},\dots,t_{j_k})  \right|.
$$
\normalsize
If $\partial_{1,\dots,k}^k f_{j_{1}\dots j_k}$ is also continuous on $[0,1]^{k}$,
\small
$$
V^{(k)}[f_{j_{1}\dots j_k}] =  \int_{[0,1]^k} \left| \partial_{1,...,k}^k f_{j_{1}... j_k}(t_{j_{1}},...,t_{j_k})  \right| dt_{j_{1}} \hspace{-1pt} \cdot \cdot \cdot dt_{j_k}.
$$
\normalsize
\end{proposition}

\section{A basis of  analytical learning theory} \label{sec:basis}

This study considers the problem of analyzing the generalization gap $\EE_\mu[L \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{S_{m}} \allowbreak [ L \hat y_{\mathcal A(S_{m})}]$ between the expected error $\EE_\mu[L \hat y_{\mathcal A(S_{m})}]$ and the training error $\hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$. For the purpose of   general applicability, our base theory analyzes a more general quantity, which is the generalization gap $\EE_\mu[L \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}]$ between the expected error $\EE_\mu[L \hat y_{\mathcal A(S_{m})}]$ and any empirical error $\hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}]$  with any dataset $Z_{m'}$ (of size $m'$) including  the training dataset with  $Z_{m'}=S_{m}$.
 Whenever we write  $Z_{m'}$, it is always including the case of $Z_{m'}=S_{m}$; i.e.,  the  case where the model is evaluated on the training set.

With our notation, one can observe  that the generalization gap is fully and deterministically specified by  a \textit{problem instance} $(\mu,S_{m},Z_{m'},L  \hat y_{\mathcal A(S_{m})})$, where we identify an omitted measure space  $(\Zcal,\Sigma,\mu)$ by the measure $\mu$ for brevity.  Indeed,  the expected error is defined by the Lebesgue integral of a function $L  \hat y_{\mathcal A(S_{m})}$ on a (unknown) normalized measure space  $(\Zcal,\Sigma,\mu)$  as
$\EE_\mu[L  \hat y_{\mathcal A(S_{m})}]  = \int_{\Zcal} L  \hat y_{\mathcal A(S_{m})} d\mu$, which is a deterministic mathematical object.
Accordingly, we introduce the following notion of  \textit{strong instance-dependence}: a mathematical object $\varphi$ is said to be  \textit{strongly instance-dependent} in  the theory of the generalization gap of the tuple $(\mu,S_{m},Z_{m'} \allowbreak, L  \hat y_{\mathcal A(S_{m})})$ if the object $\varphi$ is invariant under any  change of any mathematical  object that contains or depends on any $\bar \mu\neq \mu$, any $\hat y \neq  \hat y_{\mathcal A(S_{m})}$,  or any $\bar S_m$ such that $\bar S_{m}\neq S_{m}$ and $\bar S_{m}\neq Z_{m'}$.
Analytical learning theory is designed to provide  mathematical bounds and equations that are strongly instance-dependent. \vspace{2pt}

\subsection{Analytical decomposition of expected error} \label{sec:main_analytical_decomp}

Let  $(\Zcal,\Sigma,\mu)$ be any  (unknown) normalized measure space that defines the expected error, $\EE_{\mu}[L  \hat y]=\int_{\Zcal} L  \hat y \ d\mu$. Here, the measure space may correspond to an input-target pair as $\Zcal=\Xcal \times \Ycal$ for supervised learning,   the generative hidden space $\Zcal$ of $\Xcal \times \Ycal$ for unsupervised / generative models, or  anything else  of interest (e.g., $\Zcal=\Xcal$). Let $\mathcal T_* \mu$ be the pushforward measure of $\mu$ under a map $\mathcal T$. Let $\mathcal T(Z_{m'})=\{\mathcal T(z^{(1)}),\dots,\mathcal T(z^{(m')})\}$ be the image of the dataset $Z_{m'}$ under   $\mathcal T$. Let $|\nu|(E)$ be the total variation of a measure $\nu$ on  $E$. For vectors $a,b \in [0,1]^d$, let $[a,b]=\{t\in[0,1]^d: a \le t\le b\}$, where $\le$ denotes the product order; that is, $a \le t$ if and only if $a_j \le t_j$ for $j=1,\dots,d$. This paper adopts the convention that the infimum of the empty set is positive infinity.

Theorem \ref{thm:main} is introduced below to exploit the various structures in machine learning  through the decomposition $L  \hat y_{\mathcal A(S_{m})}(z) = (f \circ \mathcal T) (z)$ where $\mathcal T(z)$ is the output of a  representation function and $f$ outputs the associated loss. Here, $\mathcal T(z)$ can be any intermediate representation on the path from the raw
data  (when $\mathcal T(z)=z$) to the output  (when $\mathcal T(z) = L \hat y(z)$). The proposed theory holds true even if the representation $\mathcal T(z)$ is learned. The empirical error $\hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}] $ can be the training error with $Z_{m'}=S_{m}$ or the test/validation error with $Z_{m'}\neq S_{m}$.

\begin{theorem} \label{thm:main}
For  any $L  \hat y$, let $\mathcal{F}[L  \hat y]$ be a set of all  pairs  $(\mathcal T,f)$ such that  $\mathcal T:(\Zcal, \Sigma) \rightarrow ([0,1]^{d}, \mathcal B([0,1]^d))$ is a measurable function,  $f:([0,1]^{d}, \mathcal B([0,1]^d))\rightarrow (\RR, \mathcal{B}(\RR))$  is of   bounded variation as $V[f] < \infty$, and
$$
L  \hat y(z)= (f \circ \mathcal T) (z) \ \ \  \text{ for all } z \in \Zcal,
$$
where $\mathcal{B}(A)$ indicates the Borel $\sigma$-algebra on $A$.
Then, for any dataset pair $(S_{m},Z_{m'})$ (including $Z_{m'}=S_{m}$) and any $L  \hat y_{\mathcal A(S_{m})}$,
\begin{enumerate}[label=(\roman*)]
\item
$
\!
\begin{aligned}[t]
\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}]
\le \hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}] + \hspace{-3pt}  \inf_{\substack{(\mathcal T,f) \in\hat {\mathcal F}}} \hspace{-2pt} V[f] \; D^{*}[\mathcal T_{*}\mu, \mathcal T(Z_{m'})],
\end{aligned}
$

where $\hat {\mathcal F}=\mathcal{F}[L  \hat y_{\mathcal A(S_{m})}]$, and
\item
for any $(\mathcal T,f) \in\mathcal{F}[L  \hat y_{\mathcal A(S_{m})}]$ such that  $f$ is  right-continuous component-wise,
\small
\begin{align*}
\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] = \hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}]
+ \int_{[0,1]^d} \left( (\mathcal T_{*}\mu)([\bold 0,t])-\frac{1}{m'}\sum_{i=1}^{m'}\one_{[\bold 0,t]}(\mathcal T(z_{i})) \right) d\nu_{f}(t),
\end{align*}
\normalsize
where $z_i \in Z_{m'}$, and $\nu_{f}$ is a signed measure corresponding to $f$ as $f(t)= \nu_{f}([t,\bold 1])+f(\bold 1)$ and $| \nu_{f}|([0,1]^d)=V[f]$.
\end{enumerate}

\end{theorem}

The statements in Theorem \ref{thm:main} hold for each individual instance $(\mu,S_{m},Z_{m'},L  \hat y_{\mathcal A(S_{m})})$, for example, without taking a supremum over a set of other instances.  In contrast, typically in previous bounds, when asserting that  an upper bound holds on $\EE_{\mu}[L  \hat y] - \hat \EE_{S_{m}}[L  \hat y]$  for any $\hat y \in \mathcal H$ (with high probability), what it  means is  that the upper bound holds on $\sup_{\hat y \in \mathcal H}(\EE_{\mu}[L  \hat y] - \hat \EE_{S_{m}}[L  \hat y])$ (with high probability). Thus, in classical bounds including   data-dependent ones,  as   $\mathcal H$ gets larger and more complex, the bounds tend to become more pessimistic for the actual instance  $\hat y_{\mathcal A(S_{m})}$ (learned with the actual instance  $S_{m}$), which is avoided in Theorem \ref{thm:main}.

\begin{remark} \label{rem:imp_subtle}
The bound  and the equation in Theorem \ref{thm:main} are strongly instance-dependent, and in particular, \textit{invariant to  hypothesis space $\mathcal H$ and the properties of learning algorithm $\mathcal A$ over datasets different from a given training dataset $S_{m}$} (and $Z_{m'}$).
\end{remark}

\begin{remark} \label{rem:consequence_of_remark_imp_subtle}
Theorem \ref{thm:main} together with Remark \ref{rem:imp_subtle}  has an immediate practical consequence. For example, even if the true model is  contained in some  ``small'' hypothesis space $\mathcal H_1$, we might want to use a much more complex ``larger'' hypothesis space $\mathcal H_2$ in practice such that the optimization becomes easier and the training trajectory reaches a better model  $\hat y_{\mathcal A(S_{m})}$ at the end of the learning process (e.g., over-parameterization in deep learning potentially makes the non-convex optimization easier;  see \citealt{dauphin2014identifying,choromanska2015loss,soudry2017exponentially}). This is consistent with   both Theorem \ref{thm:main} and practical observations in  deep learning, although it can be puzzling from the viewpoint of previous   results  that explicitly or implicitly penalize the use of more complex ``larger'' hypothesis   spaces (e.g., see \citealt{zhang2016understanding}).
\end{remark}

\begin{remark} \label{rem:comparion_infeasible}
Theorem \ref{thm:main} does not require statistical assumptions. Thus, it is applicable even when  statistical assumptions required by statistical learning theory are violated in practice.
\end{remark}

Theorem \ref{thm:main} produces bounds that can be zero even with  $m=1$ (and $m'=1$) (as an examples are provided  throughout the paper),  supporting the concept of one-shot learning. This is true, even if the dataset is not drawn according to the measure $\mu$. This is because although  such a dataset may incur a  lager value of  $D^*$ (than a usual i.i.d. drawn dataset), it can decrease $V[f]$ in the generalization bounds of $V[f] D^{*}[\mathcal T_{*}\mu, \mathcal T(S_{m})]$.
Furthermore, by being strongly instance-dependent on the learned model $  \hat y_{\mathcal A(S_{m})}$, Theorem \ref{thm:main} supports the concept of curriculum learning~\citep{Bengio-et-al-ICML2009}. This is because   curriculum learning directly guides the learning to obtain a good model $\hat y_{\mathcal A(S_{m})}$, which minimizes $V[f]$ by its definition.

\subsection{Additionally using statistical assumption and general bounds on $D^*$} \label{sec:general-bound_star-discrepancy}

By additionally using the standard i.i.d. assumption, Proposition \ref{prop:random_sample}  provides a general bound on the star-discrepancy $D^*[\mathcal T_{*}\mu, \mathcal T(Z_{m'})]$ that appears in Theorem \ref{thm:main}.
It is a direct consequence of \citep[Theorem 2]{heinrich2001inverse}.

\begin{proposition} \label{prop:random_sample}
Let \small $\mathcal T(Z_{m'})= \{\mathcal T(z^{(1)}),\dots,\mathcal T(z^{(m')})\}=\{t^{(1)},\dots,t^{(m')}\}$ \normalsize be a set of i.i.d. random variables with values on $[0,1]^{d}$ and distribution $\mathcal T_{*}\mu$. Then, there exists a positive constant $c_{1}$ such that for all $m'\in\mathbb{N^{+}}$ and all $c_{2} \ge c_{1}$, with probability at least $1-\delta$,
$$
D^*[\mathcal T_{*}\mu, \mathcal T(Z_{m'})] \le c_{2} \sqrt{\frac{d}{m'}} $$
where $\delta=\frac{1}{c_{2} \sqrt d} (c_{1}c_{2}^{2}e^{-2c_{2}^2})^d$ with $c_{1}c_{2}^{2}e^{-2c_{2}^2}<1$.
\end{proposition}
\begin{remark} Proposition \ref{prop:random_sample} is not probabilistically vacuous in the sense that we can  increase $c_2$ to obtain $1-\delta>0$, at the cost of increasing the constant $c_2$ in the bound. Forcing $1-\delta>0$ still keeps $c_2$ constant without dependence on relevant variables such as $d$ and $m'$. This is  because $1-\delta>0$ if $c_2$ is large enough such that $c_{1}c_{2}^{2} <e^{2c_{2}^2}$, which  depends only on the constants.
\end{remark}
Using Proposition \ref{prop:random_sample}, one can immediately provide a statistical bound via Theorem \ref{thm:main} over random  $Z_{m'}$. To see how  such a result differs from that of statistical learning theory, consider the case of $Z_{m'}=S_{m}$. That is, we are looking at classic training error. Whereas statistical learning theory applies a statistical assumption to the whole object  $
\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}]  -\hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$, analytical learning theory first decomposes $\EE_\mu[L  \hat y_{\mathcal A(S_{m})}]  -\hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$ into $V[f]D^*[\mathcal T_{*}\mu, \mathcal T(S_{m})]$ and then applies the statistical assumption only to $D^*[\mathcal T_{*}\mu, \mathcal T(S_{m})]$. \textit{This makes $V[f]$   strongly instance-dependent  even with the
statistical assumption.} For example, with $f(z)=L  \hat y_{\mathcal A(S_{m})}(z)$ and $\mathcal T(z)=z$, if the training dataset $S_{m}$  satisfies the standard i.i.d. assumption, we have that with high probability,
\begin{align} \label{eq:stat_assumption_0}
\EE_\mu[L  \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}] \le c_{2} V[L  \hat y_{\mathcal A(S_{m})}]  \sqrt{\frac{d}{m}},
\end{align}
where the term $V[L  \hat y_{\mathcal A(S_{m})}]$ is  strongly instance-dependent.

In Equation \eqref{eq:stat_assumption_0}, it is unnecessary for  $m$ to approach infinity in order  for the generalization gap to go to zero. As an extreme example, if the variation of $\hat y_{\mathcal A(S_{m})}$ aligns with that of the true $y$ (i.e., $L  \hat y_{\mathcal A(S_{m})}$ is constant), we have that $V[L  \hat y_{\mathcal A(S_{m})}]=0$ and the generalization gap becomes zero even with $m=1$. This example illustrates the fact that Theorem \ref{thm:main}  supports the concept of one-shot learning via the transfer of knowledge into the resulting model $\hat y_{\mathcal A(S_{m})}$.

For the purpose of the non-statistical decomposition of $\EE_\mu[L  \hat y_{\mathcal A(S_{m})}]  -\hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$,  instead of Theorem \ref{thm:main}, we might  be tempted to  conduct a   simpler decomposition with the H\"{o}lder inequality  or its variants. However, such a simpler decomposition is dominated by a  difference between the true  measure and the empirical measure on an arbitrary set  in high-dimensional space, which suffers from the curse of dimensionality. Indeed, the proof of Theorem \ref{thm:main} is devoted to reformulating $\EE_\mu[L  \hat y_{\mathcal A(S_{m})}]  -\hat \EE_{S_{m}}[L  \hat y_{\mathcal A(S_{m})}]$ via the equivalence in the measure and the variation \textit{before taking any inequality}, so that we can avoid such an issue. That is, the star-discrepancy evaluates the difference in the measures on high-dimensional boxes with one vertex at the origin, instead of on an arbitrary set.

The following proposition proves the existence of a  dataset $Z_{m'}$  with a   convergence rate of $D^*[\mathcal T_{*}\mu, \mathcal T(S_{m})]$ that  is asymptotically faster than $O(\sqrt{1/m})$ in terms of the dataset size $m'$. This is a direct consequence of \citep[Theorem 2]{aistleitner2014low}.

\begin{proposition} \label{prop:determ_sample}
Assume that $\mathcal T$ is a surjection. Let $\mathcal T_{*}\mu$ be any  (non-negative) normalized Borel measure on $[0,1]^d$. Then, for any $m' \in \NN^+$, there exists a dataset  $Z_{m'}$ such that
$$
D^*[\mathcal T_{*}\mu, \mathcal T(Z_{m'})] \le 63 \sqrt d \frac{(2+\log_2 m')^{(3d+1)/2}}{m'}.
$$
\end{proposition}

This can be of  interest when we can choose $\mathcal T$ to make $d$ small without increasing $V[f]$ too much; i.e.,  it then provides a faster convergence rate than usual statistical guarantees. If $\Zcal \subseteq \mathbb{R}^{d_z}$ (which is true in many practical cases),   we can have $d=1$ by setting $\mathcal T: \Zcal \rightarrow [0,1]$, because there exists a bijection between $\Zcal $ and $(0,1)$. Then, although the variation of $\mathcal T$ is unbounded in general, $V[f]$  might be still  small. For example, it is still zero if the variation of $\hat y_{\mathcal A(S_{m})}$ aligns with that of the true $y$ in this space of $[0,1]$.

\subsection{General examples}

The following example  provides insights on the quality of \textit{learned} representations:

\begin{example} \label{example:hidden}

Let $\mathcal T(z)=(\phi(x), v)$ where $\phi$ is a map of any \textit{learned} representation and $v$ is a variable such that  there exists a function $f$ satisfying $L\hat y_{\mathcal A(S_{m})}(z)=f(\phi(x), v)$  (for supervised learning, setting $v:=y$ always satisfies this condition regardless of the information contained in $\phi(x)$). For example, $\mathcal \phi(x)$ may represent the output of  any intermediate hidden layer in deep learning (possibly the last hidden layer), and $v$ may encode the noise left in the label $y$. Let $f$ be a map such that $L\hat y_{\mathcal A(S_{m})} = f(\mathcal T(z))$.  Then, if $V[f]<\infty$, Theorem \ref{thm:main} implies that for any dataset pair $(S_{m},Z_{m'})$ (including $Z_{m'}=S_{m}$),
$\EE_{\mu}[L\hat y_{\mathcal A(S_{m})}] \le \hat \EE_{Z_{m'}}[L\hat y_{\mathcal A(S_{m})}] +V[f] D^*[\mathcal T_*\mu,\mathcal T(Z_{m'})].
$\end{example}

Example \ref{example:hidden}  partially supports the concept of the \textit{disentanglement} in deep learning \citep{bengio2009learning} and proposes a new concrete method to measure the  degree  of  disentanglement as follows. In the definition of \small $V[f] = \sum_{k=1}^d  \ \ \sum_{1\le j_1 <  \dots < j_k \le d} V^{(k)}[f_{j_{1}\dots j_k}]$\normalsize, each term $V^{(k)}[f_{j_{1}\dots j_k}]$  can be viewed as measuring  how \textit{entangled} the $j_{1},\dots ,j_k$-th variables are in a space of a learned (hidden) representation. We can observe  this from the definition of $V^{(k)}[f_{j_{1}\dots j_k}]$ or from Proposition \ref{prop:variation_diff} as: \small $V^{(k)}[f_{j_{1}\dots j_k}] =  \int_{[0,1]^k} \left| \partial_{1,\dots,k}^k f_{j_{1}\dots j_k}(t_{j_{1}},\dots,t_{j_k})  \right| dt_{j_{1}} \cdots dt_{j_k}$\normalsize, where \small $\partial_{1,\dots,k}^k  f_{j_{1}\dots j_k}(t_{j_{1}},\dots,t_{j_k})$ \normalsize is the $k$-th order \textit{cross} partial derivatives across the $j_{1},\dots ,j_k$-th variables.
If all the variables in a space of a  learned  (hidden) representation are completely disentangled in this sense, $V^{(k)}[f_{j_{1}\dots j_k}]=0$ for all $k\ge 2$ and $V[f]$ is minimized to $V[f]=\sum_{j_1=1}^dV^{(1)}[f_{j_{1}}]$.  Additionally, Appendices \ref{app:patho_non-flat} and \ref{app:high_deriv} provide discussion of the effect of flatness in measures and higher-order derivatives.

One of the reasons why analytical learning theory is complementary to statistical learning theory is the fact that we can  naturally combine the both. For example, in Example \ref{example:hidden}, we cannot directly adopt the probabilistic bound on $ D^{*}[\mathcal T_{*}\mu, \mathcal T(S_{m})]$ from Section \ref{sec:general-bound_star-discrepancy}, if  $\mathcal T(S_{m})$ does  not satisfy the i.i.d. assumption because $\mathcal T$ depends on the whole dataset $S_{m}$. In this case, to analyze $D^{*}[\mathcal T_{*}\mu, \mathcal T(S_{m})]$, we can  use the approaches in statistical learning theory, such as Rademacher complexity or covering number.
 To see this, consider a set $\Phi$    such that $\mathcal T \in \Phi$ and $\Phi$ is independent of $S_{m}$. Then, by applying Proposition \ref{prop:random_sample} with a union bound over a cover  of $\Phi$, we can obtain probabilistic bounds on $D^*$ with the log of  the \textit{covering number }of $\Phi$ for all representations $\mathcal T' \in \Phi$. As in data-dependent approaches (e.g., \citealt[Lemma A.9]{bartlett2017spectrally}), one can also consider a sequence of sets $\{\Phi_j\}_j$ such that $\mathcal T \in \cup_j\Phi_j$, and one can obtain a data-dependent bound on $D^{*}[\mathcal T_{*}\mu, \mathcal T(S_{m})]$ via a  complexity of $\Phi_j$.

The following example establishes the  tightness of Theorem \ref{thm:main} (i) with the 0-1 loss in general, where $\iota:\{0,1\} \rightarrow [0,1]$ is an inclusion map:
\begin{example} \label{example:loss}
 Theorem \ref{thm:main} (i) is tight in   multi-class classification with  0-1 loss as follows. Let  $\mathcal T= \iota \circ L{\hat y_{\mathcal A(S_{m})}}$. Let $f$ be an identity map.
 Then,  $V[f]=1$ and $L{\hat y_{\mathcal A(S_{m})}}(z)= (f \circ \mathcal T) (z)$ for all $z \in \Zcal$.
Then, the pair of $\mathcal T$ and $f$ satisfies the condition in Theorem \ref{thm:main} as $L{\hat y_{\mathcal A(S_{m})}}$ and $\iota$ are measurable functions. Thus, from Theorem \ref{thm:main},
$\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}]
\le V[f] D^*[\mathcal T_{*}\mu, \mathcal T(Z_{m'})]
= |(\mathcal T_{*}\mu)(\{1\})-\EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}]|)
$ (see Appendix \ref{app:proof_gexample-loss} for this derivation),
which establishes a tightness of Theorem \ref{thm:main} (i) with the 0-1 loss as follows: for any dataset pair $(S_{m},Z_{m'})$ (including $Z_{m'}=S_{m}$),
$
\left|
\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] -  \hat \EE_{Z_{m'}}[L  \hat y_{\mathcal A(S_{m})}] \right| =V[f] D^*[\mathcal T_{*}\mu, \mathcal T(Z_{m'})].
$
\end{example}

The following example applies Theorem \ref{thm:main} to a raw representation space $\mathcal T(z)=z$ and a loss space  $\mathcal T(z)=(\iota \circ L{\hat y_{\mathcal A(S_{m})}})(z)$:
\begin{example}  \label{example:1}
Consider a normalized domain $\Zcal  =  [0,1]^{d_z}$ and a Borel measure $\mu$ on $\Zcal$. For example, $\Zcal$ can be an unknown hidden generative space or an input-output space ($\Zcal=\Xcal \times \Ycal$). Let us  apply  Theorem \ref{thm:main} to this measure space with $\mathcal T(z)=z$ and $f=L\hat y_{\mathcal A(S_{m})}$. Then, if $V[L\hat y_{\mathcal A(S_{m})}]<\infty$, Theorem \ref{thm:main} implies that
for any dataset pair $(S_{m},Z_{m'})$ (including $Z_{m'}=S_{m}$) and  any $L  \hat y_{\mathcal A(S_{m})}$,
$\EE_{\mu}[L\hat y_{\mathcal A(S_{m})}] \le \hat \EE_{Z_{m'}}[L\hat y_{\mathcal A(S_{m})}] +V[L\hat y_{\mathcal A(S_{m})}] D^*[\mu,Z_{m'}]$.
\end{example}

Example \ref{example:1}  indicates  that we can regularize $V[L\hat y_{\mathcal A(S_{m})}]$ in some space  $\Zcal$ to control the generalization gap. For example, letting the model $\hat y_{\mathcal A(S_{m})}$ be invariant to a subspace that is not essential for prediction decreases the bound on $V[L\hat y_{\mathcal A(S_{m})}]$. As an extreme example, if  $x=g(y,\xi)$ with some  generative function $g$ and noise $\xi$ (i.e., a setting considered in an information theoretic approach),  $\hat y_{\mathcal A(S_{m})}$ being invariant to $\xi$ results in a smaller bound on $V[L\hat y_{\mathcal A(S_{m})}]$. This is qualitatively related to an information theoretic observation such as in \citep{achille2017emergence}.

\section{Application to linear regression} \label{sec:classical_ML}

Even in the classical setting of linear regression, recent papers (\citealt{zhang2016understanding}, Section 5; \citealt{kawaguchi2017generalization}, Section 3; \citealt{poggio2017theory}, Section 5) suggest the  need for further theoretical studies to better understand  the question of precisely   what makes a learned model generalize well, especially with  an \textit{arbitrarily} rich hypothesis space and algorithmic instability. Theorem \ref{thm:main}  studies the question abstractly for machine learning in general. As a simple concrete example, this section considers linear regression. However, note that the theoretical results  in this section can be directly applied to deep learning as described in Remark \ref{rem:linear_to_dl}.

Let $S_{m}=\{s^{(i)}\}_{i=1}^m$ be a training dataset of the input-target pairs where $s^{(i)}=(x^{(i)},y^{(i)})$. Let   $\hat y_{\mathcal A(S_{m})}=\hat W\phi(\cdot)$ be the learned model at the end of any training process. For example, in empirical risk minimization, the matrix $\hat W$ is an output of the training process,  $\hat W :=\argmin_W \hat \EE_{S_{m}} [\frac{1}{2}\|W\phi(x)-y\|_2^2]$. Here, $\phi:(\mathcal X,\Sigma_x) \rightarrow ([0,1]^{d_\phi},\mathcal B([0,1]^{d_\phi}))$ is any normalized   measurable function, corresponding to fixed features. For any given variable $v$, let $d_v$ be the dimensionality of the variable $v$. The goal is to minimize the expected error $\EE_{s} [\frac{1}{2}\|\hat W\phi(x)-y\|_2^2]$  of the learned model $\hat W\phi(\cdot)$.

\subsection{Domains  with linear Gaussian labels} \label{sec:fixed_feature_true_model}

In this subsection only, we assume that the target output $y$ is structured  such that
$
y = W^* \phi(x) + \xi,
$
where $\xi$ is a zero-mean random variable independent of $x$. Many columns of $W^*$ can be zeros (i.e., sparse) such that  $W^* \phi(x)$  uses a small portion of the feature vector $\phi(x)$. Thus, this  label assumption   can be satisfied by including a sufficient number of  elements from a basis with uniform approximation power (e.g., polynomial basis, Fourier basis, a set of step functions, etc.) to the feature vector $\phi(x)$ up to a desired approximation error. Note that we do not assume any knowledge of $W^*$.

Let $\mu_x$ be the (unknown) normalized measure for the input $x$ (corresponding to the marginal distribution of $(x,y)$). Let $X_m=\{x^{(i)}\}_{i=1}^m$ and $\tilde S_{m}=\{(x^{(i)},\xi^{(i)})\}_{i=1}^m$ be the input part and the (unknown) input-noise part of the same training dataset as $S_{m}$, respectively. We do not assume  access to $\tilde S_{m}$. Let $W_{l}$ be the $l$-th column of the matrix $W$.

\begin{theorem} \label{thm:fixed_feature_1}
Assume that the labels are structured as described above  and  $\|\hat W - W^*\|<\infty$. Then,  Theorem \ref{thm:main} implies that
\begin{align} \label{eq:fixed_feature_1}
\EE_s\left[\frac{1}{2}\|\hat W\phi (x)- y\|_2^2\right] - \hat \EE_{S_{m}} \left[\frac{1}{2}\|\hat W\phi (x)- y\|_2^2\right]
\le V[f]D^{*}[\phi_*\mu_x,\phi(X_{m})] + A_1 + A_2 ,
\end{align}
where $f(t)=\frac{1}{2}\|\hat Wt-W^* t\|_2^{2}$, $A_1 = \hat \EE_{\tilde S_{m}}[\xi^\top (\hat W-W^{*}) \phi(x)]$,  $A_2 =\EE_{\xi}[\|\xi\|^2_2]- \hat \EE_{\tilde S_{m}}[\|\xi\|^2_2] $, and
\begin{align*}
\nonumber V[f]  \le  \sum_{l=1}^{d_{\phi}} \|(\hat W_{l} - W_{l}^*)^\top (\hat W - W^*)\|_1
 + \sum_{1\le l < l' \le d_{\phi}} |(\hat W_{l} - W_{ l}^*)^\top (\hat W_{l'} - W_{l'}^*)|.
\end{align*}

\end{theorem}

\begin{remark} \label{rem:fixed_feature_1}
 Theorem \ref{thm:fixed_feature_1} is tight in terms  of both the minimizer and its value, which is explained below. The bound in Theorem \ref{thm:fixed_feature_1} (i.e., the right-hand-side of Equation \eqref{eq:fixed_feature_1}) is minimized (to be the noise term  $A_2$ only) if and only if $\hat W=W^*$ (see Appendix \ref{app:supp_rem_fixed_feature_1}   for pathological cases). Therefore, minimizing the bound in  Theorem \ref{thm:fixed_feature_1} is equivalent to minimizing  the expected error \small$\EE_s[\|\hat W\phi (x)- y\|_2^2]$ \normalsize or generalization error (see Appendix \ref{app:supp_rem_fixed_feature_1} for further  details). Furthermore, the bound in Theorem \ref{thm:fixed_feature_1} holds with equality if  $\hat W=W^*$. Thus, the bound is tight in terms  of the minimizer and its value.
\end{remark}

\begin{remark} \label{rem:fixed_feature_1_noize-term}
For $D^{*}[\phi_*\mu_x,\phi(X_{m})]$ and $A_2$, we can straightforwardly  apply the probabilistic bounds under the standard i.i.d. statistical assumption. From Proposition \ref{prop:random_sample},
with high probability, $
D^{*}[\phi_*\mu_x,\phi(X_{m})] \le O (\sqrt{d_{\phi}/m}).
$
From Hoeffding's inequality with  $M \ge \|\xi\|^2_2$,  with probability at least $1-\delta$,  $A_2\le M\sqrt{\ln(1/\delta)/2m}$.
\end{remark}

It is not necessary for  $D^{*}[\phi_*\mu_x,\phi(X_{m})]$  to approach  zero to minimize the expected error; irrespective of whether the  training dataset satisfies a certain  statistical assumption to bound $D^{*}[\phi_*\mu_x,\phi(X_{m})]$, we can minimize the expected error via making $\hat W$ closer to $W^*$ as shown in Theorem \ref{thm:fixed_feature_1}.

\subsection{Domains  with unstructured/random labels} \label{sec:fixed_feature_no_struc}

In this subsection, we discard the linear Gaussian label assumption in the previous subsection and consider the worst case scenario where
$y$ is a variable independent of  $x$.
This corresponds to the random label experiment by \citet{zhang2016understanding}, which posed another  question: how to theoretically distinguish the generalization behaviors with  structured labels from those with  random labels. Generalization behaviors in practice are expected to be  significantly different in problems with structured labels or random labels, even when the hypothesis space and learning algorithm  remain unchanged.

As desired, Theorem \ref{thm:fixed_feature_2} (unstructured labels) predicts a completely different generalization behavior from that in Theorem \ref{thm:fixed_feature_1} (structured labels), even with an identical hypothesis space and learning algorithm. Here,
we consider the normalization of $y$ such that  $y \in [0,1]^{d_y}$. Let $\mu_s$ be the (unknown) normalized measure for the pair $s=(x,y)$.

\begin{theorem} \label{thm:fixed_feature_2}
\emph{}
Assume  unstructured labels as described above. Let $M=\sup_{t\in[0,1]}\|\hat Wt - y\|_{\infty}$. Assume that $\|\hat W\|<\infty$ and $M<\infty$. Then, Theorem \ref{thm:main} implies that
\begin{align} \label{eq:fixed_feature_2}
\EE_{s}\left[\frac{1}{2}\|\hat W\phi (x)- y\|_2^2\right] - \hat \EE_{S_{m}} \left[\frac{1}{2}\|\hat W\phi (x)- y\|_2^2\right]
 \le V[f]D^{*}[\mathcal T_*\mu_{s}, \mathcal T(S_{m})],
\end{align}
where  $\mathcal T(s) = (\phi(x),y)$, $f(t,y)=\frac{1}{2}\|\hat Wt-y\|_2^2$, and
$$
V[f] \le (M+1) \sum_{l=1}^{d_\phi}\|\hat W_{l}\|_1 +\sum_{1\le l<l' \le d_\phi} |\hat W^\top_{l} \hat W_{l'}|+ d_{y} M.
$$
\end{theorem}

Unlike in the structured case (Theorem \ref{thm:fixed_feature_1}),  minimizing the bound on the generalization gap in the unstructured case requires us to control the norm of $\hat W$, which corresponds to the traditional results from statistical learning theory. As in statistical learning theory, the  generalization gap in Theorem \ref{thm:fixed_feature_2} (unstructured labels)  goes to zero as $D^{*}[\mathcal T_*\mu_{s}, \mathcal T(S_{m})]$ approaches zero via certain statistical assumption: e.g., via Proposition \ref{prop:random_sample}, with high probability, $D^{*}[\mathcal T_*\mu_{s}, \mathcal T(S_{m})]\le O (\sqrt{(d_{\phi}+d_y)/m})$.  This is in contrast to Theorem \ref{thm:fixed_feature_1} (the structured case) where we require no statistical assumption for the generalization gap to approach zero within polynomial sample complexity.

\begin{remark} \label{rem:linear_to_dl}
(Theorems  \ref{thm:fixed_feature_1} and \ref{thm:fixed_feature_2}  on representation learning)
\textit{Theorems  \ref{thm:fixed_feature_1} and \ref{thm:fixed_feature_2}  hold true,  even with learned representations $\phi$, instead of fixed features}. Let $\mathcal \phi(x)$ represent the last hidden layer in a neural network or the learned representation in representation learning in general. Consider the squared loss (square of output minus target). Then, the identical proofs of Theorems  \ref{thm:fixed_feature_1} and \ref{thm:fixed_feature_2} work with the learned representation $\phi$.
\end{remark}

\section{From analytical learning theory to methods in deep learning} \label{sec:method}
This section  further demonstrates the practical relevance of  analytical learning theory by showing its application to derive empirical methods. The complete code of our method and experiments is publicly available at \url{https://github.com/Learning-and-Intelligent-Systems/DualCutout}.

\subsection{Theory}
We consider multi-class classification with a set $Y$ of class labels. Then,

\begin{align*}
&\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] - \hat \EE_{S_m}[L  \hat y_{\mathcal A(S_{m})}]
\\ &= \sum_{y \in Y} p(y) \EE_{\mu_{x|y}} [L  \hat y_{\mathcal A(S_{m})}]-  \hat p(y) \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}] \pm    p(y) \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}]
\\ & = \sum_{y \in Y} p(y)\left( \EE_{\mu_{x|y}} [L  \hat y_{\mathcal A(S_{m})}]-    \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}] \right) + \left(p(y)-\hat p(y) \right)  \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}],
\end{align*}
where $\hat p(y) \triangleq \frac{|S_{x|y}|}{m}$, $ \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}] \triangleq \frac{1}{|S_{x|y}|} \sum_{x \in S_{x|y}} L  \hat y_{\mathcal A(S_{m})}(x,y)$, and $S_{x|y} \subseteq S_m$ is the set of the training input points $x$ of the label $y$. Within the sum over $y$, by applying Theorem 1 (i) to each first term and Hoeffding's inequality to each second term, we have that with probability at least $1-\delta/2$,
\begin{align*}
&\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] - \hat \EE_{S_m}[L  \hat y_{\mathcal A(S_{m})}]
\\ &\le \sum_{y \in Y} p(y)\inf_{\substack{(\mathcal T_{y},f_{y}) \in\hat {\mathcal F_{y}}}} \hspace{-2pt} V[f_{y}] D^{*}[(\mathcal T_{y})_{*}\mu_{x|y}, \mathcal T_{y}(S_{x|y})] +  \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}] \sqrt{\frac{\log 2/\delta}{2|S_{x|y}|}}.
\end{align*}

Assume that there exists a generative (unknown) hidden space $G:(y,z) \mapsto x$ where the true label of the input $x=G(y,z)$ is  $y$ for any $z$ in the its normalized domain. We now set $\mathcal T_{y}:(x,y) \mapsto z$ and $f_y:z\mapsto L  \hat y_{\mathcal A(S_{m})}(x,y)$ where $z$ is the \textit{unknown}  hidden space that does not affect the true label. This choice does not depend on the dataset although it is unknown. Thus, by applying Proposition 2 with these $(T_{y},f_y)$, we have that with probability at least $1-\delta$,
\begin{align} \label{eq:method}
\EE_{\mu}[L  \hat y_{\mathcal A(S_{m})}] - \hat \EE_{S_m}[L  \hat y_{\mathcal A(S_{m})}] \le\sum_{y \in Y}  c_{2}p(y)V[f_{y}] \sqrt{\frac{d_{z}}{|S_{x|y}|}}+  \hat \EE_{S_{x|y}}[L  \hat y_{\mathcal A(S_{m})}] \sqrt{\frac{\log 2/\delta}{2|S_{x|y}|}},
\end{align}
where $d_z$ is the dimensionality of the generative hidden space of $z$ and $c_2$ is a constant defined in  Proposition 2.

Equation \ref{eq:method} tells us that if $V[f_{y}]$ is  bounded by a constant, the generalization error  \textit{goes to zero in polynomial sample complexity}  \textit{even with an arbitrarily complex hypothesis space and non-stable learning algorithm.} If the loss is 0-1 loss,   $V[f_y]=0$ when $(y,z)\mapsto   \hat y_{\mathcal A(S_{m})}(x)$ is invariant over $z$. In other words, to control $V[f_y]$, we want to have a model that is more invariant over the  space of $z$, which intuitively makes sense.

\subsection{Methods}

The above result provides a theoretical basis for a family of \textit{consistency-based regularization} methods, including $\Pi$-Model ~\citep{laine}, virtual adversarial training ~\citep{takeru} and regularization with stochastic transformations and perturbations ~\citep{sajjadi}. These consistency-based regularization methods have been empirically   successful heuristics. These  methods are based on the intuition that perturbations of a data point $x \mapsto\ \tilde x $ should not change the output of a  model as $\hat y(x) \approx \hat y( \tilde x)$ if the true label is invariant under the perturbation; i.e., $y^*(x)=y^*(\tilde x)$ where $y^{*}$ outputs a correct label.
This intuitive goal is achieved by minimizing  $d(\hat y(x), \hat y(\tilde x))$ with respect to the trainable model  $\hat y$, where $d(, )$ measures a distance between the two outputs. In Equation \ref{eq:method}, these  methods can be viewed to control $V[f_{y}]$ by making the model $\hat y$ more invariant over the  space of $z$. Therefore, our theory formalizes the intuition of these regularization methods in terms of the generalization gap.

In order to more effectively minimize the bound on the generalization gap in Equation \ref{eq:method}, we propose a new regularization method, called \textit{dual-cutout}. For each training input $x$, our dual-cutout method  minimizes the following   regularization loss $\ell_{\text{reg}}(x, \theta)$ with respect to $\theta$ (in addition to the original classification loss):
$$
\ell_{\text{reg}}(x,\theta)= \int_{(x_{1},x_{2})} \|h(x_{1},\theta) - h(x_{2},\theta)\|_2 ^2 dP(x_{1}, x_{2}|x),
$$
 where $h(x',\theta)$ is the post-softmax output of the last layer of  a neural network with  parameters $\theta$ (given an input $x'$),  and $(x_{1}, x_{2}) \sim P(x_{1}, x_{2}|x)$ are two randomly sampled inputs of two random cutouts of a given natural input $x$. Here, we set $P(x_{1}, x_{2}|x)=P(x_{1}|x)P(x_{2}|x)$, and $P(x_{1}|x)=P(x_{2}|x)$ is the probability distribution over random cutout input $x_{1}$ given a original (non-cutout) input $x$; i.e., $P(x_{1}|x)=P(x_{2}|x)$ represents the same random cutout procedure as  single-cutout method  in the previous paper \citep{cutout}.  As this additional regularization loss gets smaller, the model becomes more insensitive over the  hidden space of $z$, implicitly minimizing $V[f_{y}]$ and the bound on the generalization gap in Equation \ref{eq:method}.

Table \ref{tbl:method} compares the test error of dual-cutout against single-cutout and the standard method for three benchmark datasets, namely CIFAR10, CIFAR100 and SVHN. Dual-cutout outperforms baseline methods by a significant margin.

\begin{table}[h!]
\centering
\caption{Test error (\%) with different regularization methods.   } \label{tbl:method}
\begin{tabular}{lcccc}
\toprule
Method & CIFAR-10 & CIFAR-100 & SVHN \\
\midrule
Standard        & 3.79 $\pm$ 0.07 & 19.85 $\pm$  0.14 & 2.47 $\pm$ 0.04 \\
\midrule
Single-cutout  & 3.19 $\pm$ 0.09 & 18.13 $\pm$ 0.28 &  2.23 $\pm$ 0.03  \\
\midrule
Dual-cutout    & 2.61 $\pm$ 0.04 & 17.54 $\pm$ 0.09 &  2.06 $\pm$ 0.06 \\
\bottomrule
\end{tabular}
\end{table}

We conducted all the experiments with the WideResNet$28\_10$ ~\citep{wrn} architecture and report the test errors at the end of   300 training epochs. We used SGD with the learning rate 0.1 and the momentum 0.9. At each step of SGD, to minimize the regularization loss $\ell_{\text{reg}}(x, \theta)$ of dual-cutout, we used the sampled gradient $\nabla_\theta \|h(x_{1}, \theta) - h(x_{2}, \theta)\|_2^2$ where $(x_{1}, x_{2})$ is sampled as $(x_{1}, x_{2}) \sim P(x_{1}, x_{2}|x)$. The learning rate was annealed at epochs 150 and 225 by a factor of 0.1. We used standard data-augmentation and preprocessing for all the datasets. For each dataset, we choose the  cutout size as reported in ~\citep{cutout}. We performed five trials of each experiment and report the standard deviation and mean of test error in Table ~\ref{tbl:method}.

\section{Discussion} \label{sec:beyond_classical_ML}

\begin{table*}[b!] \footnotesize
\centering
\caption{A simplified comparison, wherein GG denotes the generalization gap} \label{tbl:slt-vs-alt}
\begin{tabular}{l | c | c}
\noalign{\hrule height 0.9pt}
& Statistical Learning Theory  & Analytical Learning Theory   \\  \noalign{\hrule height 0.4pt}
GG is characterized by  & hypothesis spaces  $\mathcal H$ or algorithms $\mathcal A$  & a learned model $\hat y_{\mathcal A(S_{m})}$   \\
GG is decomposed via & statistics  & measure theory \\
Statistical assumption & is required  & can be additionally used \\
Main focus is when &  a (training) dataset $S_{m}$ remains random &  a (training) dataset $S_{m}$ is given
\\
Bounds on GG are  & not strongly instance-dependent & strongly instance-dependent\\
\noalign{\hrule height  0.9pt}
\end{tabular}
\end{table*}

Table \ref{tbl:slt-vs-alt} summarizes the major simplified differences between statistical learning theory and  analytical learning theory. Because of the differences in the assumptions and the objectives, the proposed learning theory is not directly comparable in terms of sample complexity against previous learning theory. Instead of focusing on comparable sample-complexity, analytical learning theory focuses on complementing previous learning theory by providing additional practical insights.
Indeed, the real-world phenomena that are analyzed are different in statistical learning theory and analytical learning theory.
Typically in statistical learning theory,  some upper bound holds  over  a fixed   $\mathcal H$ or  a fixed   $\mathcal A$  \textit{with high probability over different random datasets}. In contrast, in analytical learning theory, some upper bound holds \textit{individually for each problem instance}.

An another difference between  statistical learning theory and  analytical learning theory lies in the property of strong instance-dependence. Any generalization bound that depends on a non-singleton hypothesis space $\mathcal H \neq \{\allowbreak \hat y_{\mathcal A(S_{m})}  \}$, such as ones  with Rademacher complexity and VC dimension, is \textit{not} strongly instance-dependent because the non-singleton hypothesis space contains $\hat y \neq  \hat y_{\mathcal A(S_{m})}$, and the bound is not invariant  under an arbitrary change of $\mathcal H$. The definition of  stability  itself depends on  $\bar S_m$ that is not equal to $S_{m}$ and $ Z_{m'}$ \citep{bousquet2002stability}, making the corresponding bounds  be  \textit{not} strongly instance-dependent. Moreover, a generalization bound that depends on a concept of  random  datasets   $\bar S_m$ different from  $S_{m}$ and $ Z_{m'}$ (e.g.,  an additive term $O(\sqrt{1/m})$ that measures a deviation  from an expectation over  $\bar S_m\neq S_{m},Z_{m'}$) is  \textit{not} strongly instance-dependent, because the bound is not invariant under an arbitrary change of $\bar S_m$.

Data dependence  does not imply  strong instance-dependence.
For example, in the data-dependent bounds of the luckiness framework \citep{shawe1998structural,herbrich2002algorithmic},   the definition of $\omega$-smallness of the luckiness function  contains a non-singleton hypothesis space  $\mathcal H$, a  sequence of non-singleton hypothesis spaces  (ordered in a data-dependent way by a luckiness function), and a supremum over $\mathcal H$ with the probability over  datasets $\bar S_m \neq S_{m}$ (with $ Z_{m'}=S_{m})$ (e.g., see Definition 4 in \citealt{herbrich2002algorithmic} with contraposition).
As exemplified in the luckiness framework,  one can usually turn   both data-dependent and data-independent bounds  into  more data-dependent ones by considering a sequence of hypothesis spaces or sets of learning algorithms. However, such data-dependent bounds still contain the complexity of a non-singleton hypothesis space (and  dependence on the definition of the sequence). The data-dependent bounds with empirical Rademacher complexity
\citep{koltchinskii2000rademacher,bartlett2002model} also depend on a non-singleton hypothesis space and its empirical Rademacher complexity. Moreover, the definition of robustness itself depends on   $\bar S_m$, which is not equal to $S_{m}$ or $ Z_{m'}$
\citep{xu2012robustness}. Therefore, all of these data-dependent bounds are not strongly instance-dependent.

The fact that Theorem \ref{thm:main}  is invariant to  the complexity of hypothesis space $\mathcal H$ and certain details of a learning algorithm $\mathcal A$ can be both advantageous and disadvantageous, depending on the objective of the analysis. As we move towards the goal of artificial intelligence, $\mathcal H$ and $\mathcal A$ would become extremely complex,  which can pose a challenge in theory. From this  viewpoint, analytical learning theory can also be  considered as a methodology to avoid such a challenge, producing  theoretical insights for intelligent systems with arbitrarily complex $\mathcal H$ and $\mathcal A$, so long as other conditions are imposed on the actual functions being computed by them.

\acks{We gratefully acknowledge support from NSF grants 1420316, 1523767 and 1723381,  from AFOSR FA9550-17-1-0165, from ONR grant N00014-14-1-0486, and from ARO grant  W911 NF1410433, as well as support from NSERC, CIFAR and Canada Research Chairs. Vikas Verma was supported by Academy of Finland project 13312683 / Raiko Tapani AT kulut. }

redpajama_set_name: RedPajamaArXiv


text:
The Benito Juárez Hemicycle is a Neoclassical monument located at the Alameda Central park in Mexico City, Mexico and commemorating the Mexican statesman Benito Juárez. The statue of Juárez is flanked by marble Doric columns. There are two allegorical female statues next to Juárez, representing the fatherland and law. The pedestal bears the inscription "Al benemerito Benito Juárez la Patria" (Spanish: "To the meritorious Benito Juárez, the Homeland"). It was depicted on the reverse of the 20 peso bill of Series C and D, issued in 1994.

History
The construction began in 1906 it mark the centennial of Juárez's birth. The engineers for the construction of the monument were assigned by Porfirio Díaz. Statues were sculpted by the Italian artist Alessandro Lazzerini.

The monument was dedicated on September 18, 1910.

Style
It is Neoclassical style, semicircular, with strong Greek influence; it has twelve Doric columns, supporting an entablature and frieze structure of the same order.
On both sides it has two golden spikes.

At the center is a sculpture composed of Benito Juarez seated with two allegories: one representing the homeland crowning Juarez with laurels in the presence of a second that represents the law in the basement has festoons, another sculpture center that chairs a republican eagle with open wings in a facing, with neoaztec frets, which lie two lions.

On the central pillar there is a medallion surrounded by a laurel, inscribed with the following;

References

1910 establishments in Mexico
1910 sculptures
Alameda Central
Allegorical sculptures
Buildings and structures completed in 1910
Monuments and memorials in Mexico City
Neoclassical architecture in Mexico
Outdoor sculptures in Mexico City
Sculptures of men in Mexico
Sculptures of women in Mexico
Statues in Mexico City
Benito Juárez

redpajama_set_name: RedPajamaWikipedia


text:
Brown University Students Overwhelmingly Vote in Favor of Reparations for Black Students
April 6, 2021 April 5, 2021 Georgia Star News Staff
by Eric Lendrum
On Monday, Students at the Ivy League school Brown University voted in favor of two resolutions approving reparations for black students, as reported by the Washington Free Beacon.
Both resolutions seek to identify any black students who are direct descendants of slaves, or "who were entangled with and/or afflicted by the University and Brown family and their associates," in reference to the university's founder Nicholas Brown Jr.
One resolution would give priority admission to any such black students, while the other would give direct monetary payments to said students. In the vote amongst all students on campus, the admissions resolution received 89 percent of the vote, while the financial payment resolution received 85 percent. The vote was held after the student government at Brown passed a resolution, introduced by the student government president Jason Carroll, "calling upon Brown to attempt to identify and reparate the descendants of slaves entangled with the university."
Brown is only the latest university to approve such race-based measures. Georgetown University approved its own plan in 2019 to identify and pay reparations to students who are identified as being descendants of slaves. And a spokesman for Brown University implied that the school is not done with so-called "anti-racism" measures, telling the Beacon that "the current work of Brown's Task Force on Anti-Black Racism will make recommendations on more Brown can do to address the legacy of slavery."
And yet, in his time, Nicholas Brown Jr. was a vocal opponent of slavery and an active abolitionist. One of his descendants, historian Sylvia Brown, voiced her opposition to the reparations plan and pointed out that her ancestor explicitly prohibited the captains of his merchant ships from trafficking slaves, as well as forbidding the family's Providence Bank from lending to slave traders.
Eric Lendrum reports for American Greatness.
Photo "Brown University" CC 3.0.
Content created by the Center for American Greatness, Inc. is available without charge to any eligible news publisher that can provide a significant audience. For licensing opportunities for our original content, please contact [email protected].
National, Newsanti-black racism, black students, Brown University, Georgetown University, higher education, Racism, Slavery, university students
Commentary: U.S. China Relations Go from Bad to Worse
Commentary: Too Much of a Unity
Commentary: Joe Biden, the Deep State Puppet
U.S. Energy Department Spent over $1 Billion on Failed Carbon-Cutting Projects
GOP Lawmakers Demand Ouster of Education Secretary over Reported Role in Targeting Parents

redpajama_set_name: RedPajamaCommonCrawl


text:
At CodeMedia 360, we provide tools and insights to help guide your business to success.
ThrowBack Thursdays: Silicon Valley's Humble Beginnings
In In the News, Throw Back ThursdaysTags Silicon Valley, throwback thursdays March 16, 2017 106 Views codemedia360
In Palo Alto there is a house with a plaque that labels it as the birthplace of electronics. It was in that house in 1912 that Lee de Forest and his collaborators first successfully utilized the vacuum tube to amplify sound. This technology became the basis for radio, radar, television, tape recorders and ultimately electronic computers. De Forest and his associates were working for the Federal Telegraph Co. which had been established in Palo Alto in 1909 by people associated with Stanford University.
The first commercial radio broadcast were from downtown San Jose at San Fernando and First Street. Later, in 1927, the first all-electronic transmission of television was achieved in San Francisco.
But the broadscale development of the electronics industry would not come until much later. The key figure in this development was Frederick Terman.
Terman did his undergraduate work at Stanford but went to M.I.T. for his Ph.D. in electrical engineering. He returned to Stanford to join the faculty and in 1925 was teaching a course in radio engineering. Terman worked to build up Stanford's program in electronics and electrical engineering. He was successful but the after receiving a topnotch education at Stanford graduates had to move to the East Coast to find suitable jobs.
In 1937 Terman began encouraging Stanford faculty and graduates to start businesses locally. The first and most famous firm founded in this way was Hewlett-Packard. H-P was founded on the basis of an audio-oscillator invented by William Hewlett under the guidance of Terman. David Packard was a former faculty member of Stanford whom Terman encouraged to work with Hewlett on the development of the audio-oscillator. One of the first big contracts of Hewlett-Packard (HP) was to provide the audio-oscillators used by the Disney Corporation in making its classic cartoon movie, Fantasia.
Another firm founded with the encouragement of Terman was Varian Brothers. The product, in this case, was a device for generating microwaves that could be used in radar, among other things.
Terman created an industrial park on 660 acres of Stanford's land where the land was leased to electronics and other high technology companies on long-term leases. Hewlett-Parkard and Varian Brothers were among the first tenants. This provided income to Stanford and created an agglomeration in high tech industry. At first industry came to Palo Alto to have access to the expertise at Stanford. Later the level of technical expertise in industry in the area exceeded the academic expertise of Stanford. At that point the electronic industry expanded to cities near to Palo Alto, first Mountain View and later Sunnyvale.
The Semiconductor Industry
After leaving Fairchild Semiconductor, Robert Noyce, Gordon Moore and Andrew Grove founded Intel in 1968 to produce integrated circuits. Orginally Intel's major business was supplying memory chips for mainframe and minicomputers. Now its major business is producing microprocessors which are the central processing units of microcomputers.
The Silicon Valley's output is comparable to a major country, such as the New Zealand ($130.7 billion in 2008), but not quite so great as that of Singapore ($181.9 billion in 2008).
In the early 1970s, professor Vinton Cerf, known as the "father of the Internet," developed with a colleague the TCP/IP protocols which would become the standard for Internet communication between computers. In the 1980s, John Cioffi and his students realized that traditional phone lines could be used for high-speed data transmission, leading to the development of digital subscriber lines (DSL). In 1991, SLAC physicist Paul Kunz set up the first web server in the U.S. after visiting Tim Berners-Lee, creator of the World Wide Web, in Geneva, Switzerland.
The Internet, of course, is central to the story of Silicon Valley. Google, the web's most popular search engine and one of the world's most influential companies, got its start at Stanford when Sergey Brin and Larry Page developed their page rank algorithm as graduate students in the 1990s. Before them, alumni Jerry Yang and David Filo founded Yahoo. Other legendary Silicon Valley companies with strong ties to Stanford include Cisco Systems, Hewlett-Packard Company, Intuit, Silicon Graphics, and Sun Microsystems.
Read Full Articles Below
http://www.sjsu.edu/faculty/watkins/sivalley.htm
https://www.stanford.edu/about/history/history_ch3.html
codemedia360
What You Need To Know Before Selling Online!
Sessions: 360 – How To Add Images To an HTML or WordPress Site Using Photoshop
29 Common Google Analytics Data Errors And How To Fix Them
Luxury Brands Are Becoming Big Players In The E-Commerce Game
Why Did Google Have So Much Success
Yelp 5 Star Business
Online Store Tips: Does Your Platform Make You Angry?
Online Store: Are Tasks Taking Too Much Time To Get Done?
ThrowBack Thursday: Code We Use Today, What Are They And Who Created Them?
LevelUp Commerce
Sessions 360
#TBT Adobe Flash blogger Burbank Chamber of Commerce Consumer Research Customer Service Ecommerce 2020 Efficiency Free Advice google Google Analytics Happy Valentines Day History of Electronics History of the Internet Internet Statistics In The News Landing Pages LevelUp Commerce Luxury Brands Minimum Viable Product Online Marketing Online Store page ranking Photoshop Tutorials Plugins Presidence Day Promotions Quality Content Security SEO sessions 360 Silicon Valley Small Business templates themeforest throwback thursdays Tips and Tricks Updates User Experience Web Application Web Design Web Tutorials WordPress WordPress Tutorials yoast SEO
Who you are?
Startup Business Enterprise
Website Software Mobile App
$1K - $5K $10K - $50K $50K+
Telephone: 310-775-1883, E-mail: info@codemedia360.com
Everything At Your Fingertips


Accordingly, you wont have the possiblity to describe why you've got contained something special on your restart, and so the reviewers might have to count upon the written sentence with no excess editorial. It is wise to learn you have gained from our own experience. They ought to offer concrete examples together with detailed explanations.
Let's say buy coursework help you'd prefer to improve the overall look of your kitchen. Only decide on a few sheets daily, and have everybody in the class try to guess that is being described. The artist or author with composed the audio determines which type of license it'll possess, and therefore, the bills associated.
You are ready to check the way the record is being left. In case your document will most likely have tons of pictures, then you may create individual connections to hold your images. The template can be seen within PSD format.
Still another instance where they're required is, once https://www.aare.edu.au/blog/?p=922 you might need to sell home. The very last advantages and references perform with the most considerable role in developing the perfect creation of the paper. You're ready to talk about your specific requirements having a designer and so they're going to design your own pay to your specific needs.
Infographic-style aids are becoming popular, and they are some times effective when performed effectively. The fantastic checking instruments of clinical tests on the web aren't really challenging to discover any more. If you should be seeking cheap dissertation assistance services, then then you may see dozens and lots of sites about the web that you're ready to look out for there.
So in case the provider wishes someone using a background in Python and PHP coding, they can get in touch using the recruiters. Choose each item you'd really like to modify, and following that click or tap touse the exact modifications to all these items in 1 go. Open sans can secure the business done nicely as an alternate font alternative.
Whenever you get a resume, you will need to be aware which you're receiving a worthwhile result since you do not require a buy custom written resume that's maybe not really worth the paper it's written on. While investing in a box of restart paper, you need to make sure it's the very proper match for you personally and your work history. If you select the template wisely, it's feasible to own a standout resume that's distinctively yours.
Your study area doesn't even will need to become silent. A lot of folks are not knowledgeable about the four key types of documents. You can find dozens and dozens of sites available using a huge number of songs to pick from.
On top of that, employing to enmu is wholly free of charge. Purchase a trip for their own site Dropbox is among the good file hosting service I have been employing to get long instant. Start with creating a brief collection of what you're confident you are ready to execute.
Individuals often underestimate the effect of tiny persistent activities, therefore they wind up accomplishing practically nothing rather than. Beneficial should you merely wish to grow into a indication of one's data within a professional way. The key significance of the research paper would be always to deliver exactly the accurate effects and also useful solution of the situation.
When it has to do with all the text, or copy, of one's trip booklet, becoming descriptive is vital. Academic composing could become described as a hard and tiring endeavor to finish, but if you understand the optimal/optimally location to acquire professional aid, you wont struggle. A movie essay is truly a set of images using an total theme or theme.
Once you buy a resume, you have to be aware that you are receiving a rewarding outcome since you do not require a buy custom written resume that's maybe not worth the paper it's written on. If buying a box of restart paper, you should make certain it's the very best fit for you and work history. In the event you pick the template sensibly, it's feasible to possess a whole standout resume that's uniquely yours.
For students, there's a plethora of complimentary digital tools which generate composing and writing their essays and reports simpler and faster. 1 additional great template among the checklist is known as Flux. It's not straightforward to know who might help write my faculty paper for the money, or produce my own paper at no cost, which may not be a prudent idea.
Continue until only two individuals are left . Let's have a closer look in what type of service that you have to pick from all which can be readily available. If you like interacting with cuttingedge technological innovation and receiving others excited about the most recent services and products, you may possibly be described as considered a prime prospect to get a project to get a Best Purchase sales associate.
What Is So Fascinating About Best Buy Resume Paper?
Still another instance where they've been required is, once you might need to to sell home. The very last advantages and references play with the appreciable part in acquiring the best creation of the newspaper. When you've set your purchase, the authors that are interested and proficient at the okay area commence bidding because of this.
To begin with, you will need to concentrate on the subject of issue. Know which you've got to be extended a score of at least 145 on just about every single topic in order to go. It's mandatory that you believe you should create your own personal content, also a thing which you will soon be able position to phone your personal.
Let's imagine you'd like to improve the overall look of your own kitchen. Randomly decide on a couple sheets daily, and have everybody at the class attempt to figure that will be described. For others, you will need to produce an in-person presentation.
Resume Dictionary describes four distinct kinds of resumes. The outcome will probably also inform you precisely what topics and subjects you have to work more. Because the GED practice exams can be found the web, you'll be able to elect to study wherever provided you've got an on-line relationship.
Pre-position check on the internet is a easy and exemplary solution that can knock out of your proofreading woes. Purchase a visit to their site Dropbox is among the superior record hosting agency I Have been employing for long minute. Begin by building a quick collection of what you're confident you are able to do.
The issue together with scanned documents is they are simply images of paper records. If your paper will probably have plenty of images, you may create individual connections to record your images. Applying Lucidpress, then you can choose a template that fulfills your vacation location and personalize it in whatever manner you'd like.
ORGANIZATIONAL SKILLSUnless you're requesting for a job as a scientist that is mad, organization is a significant talent for almost any work. To start out with, our expert team has generated an exceptional sentence structure on-line fixer that can assist you in rectifying all the mistakes in thesis newspapers. Apply interviews with an alternative person, this means that you may learn how to remain trendy when responding for some other individual's opinions.
In case a organization can possibly be thought of as a tourist spot, we'll explain just how exactly to generate a journey brochure at no price. That which you also need to complete is understand that not every product is right for everybody and also you've got to have the ability to tell an individual they don't really require any of your own goods. A couple of the templates are all free of charge while some charge a number dollars.
Only presume the restart we'll write for you personally can alter all of your career! You simply require look at our recommendations to determine which we're the ideal organization to pick when you want to buy restart papers. At the consumer inspection, you also ought to be more in a place to grasp exactly what their needs are to get social media marketing along with the way you're ready to aid them.
The ideal research papers aren't written readily even when you're an expert author. You may search for a book club that's already begun, or you can commence your personal. You may have to publicize your publication aggressively.
You're able to check the way in which the record is being left handed. Fortunately it is quite straightforward to guard your device with disc security. The template are found within PSD format.
Usually, a work seeker finds herself or himself at a pickle simply because they've held lots of distinct positions as time passes and don't know how exactly to hold the restart focused for a certain placement. Sooner or later, it is far superior to learn the full draft again and fix the faults in place of doing it in the start. For the optimum outcomes, you would like to concentrate on paraphrasing for averting the plagiarism precisely.
According to Outread, you ought to be at a place to allow it to be during your articles more quickly than customary without a loss of understanding. There are a lot of handy tips which may aid you in modifying the draft correctly. You'll none the less require some specialist enhancing, however, it goes to cost you a terrific deal less.
It is possible to change the colors if you want. There's a structure and a procedure to learn factors. You begin harnessing the ability of experiencing the capacity to produce your reality.
1 other great difficulty is which you simply discover that it's feasible to supply a great deal of data in an easy and trendy method. When businesses elect to create the electronic switch, they do not just lower deforestation and greenhouse gas emissions, however, in addition they increase their reputation in the market. As luck would have it,'s a whole lot of those.
Proofreading programs really are astoundingly convenient particularly to students which are continuously trying to fulfill deadlines. The outcomes will probably even tell you just what subjects and topics you have to focus more. A laboratory pc computer software may last to keep the exam records along side the patient record.
Based in your demographic, you could be able to track down a book club by means of an organization or institution you're already some of. To make you comfortable with all the complete encounter, we've designed a stay chat, and that means you may stay in touch with your writer during the job strategy. Don't hesitate to cite issues inside a team you'd to overcome.
Another superb characteristic that it is likely to use is custom critical binding for each and every sort of cell . You are able to uncover the material analyzed if you prefer to realize exactly what mold you've got. Certainly, it's going to have a while to be embraced by means of a wide array of smart devices.
The technology merchandise, expertise and service you've got to earn your business effective. All of the very best with your work program, perfect acquire phd study proposal for just about any program. You will notice a range of those templates are free of charge to make use of and many others need a premium accounts.


text:
From the Top End of Down Under, it's a new year with Dr. Starkey.
By Kathryn Starkey, MD, and Molly Evans
G'day from the Top End,
Over 60,000 years ago, when the sea level was much lower, people walked from New Guinea to Northern Australia and these wanderers were the origin of the Australian Indigenous Aborigines. By the time Europeans arrived in 1788, the Aborigines—scattered throughout the continent as hunters and gatherers—may have numbered close to a million people. They are the oldest continuously maintained culture on earth and now number about 455,000. However, homelessness is highly visible among Aboriginal people.
We witnessed this first hand while working in a small remote 60-bed hospital in an outback town called Katherine (where 2011 Tour de France champion Cadel Evans hails from). I worked alongside an outreach nurse specialist who provided gynecology care to our patients, whom she called "the women from the long grass". Often found living in the surrounding fields, these women lack education and have nowhere else to live. The government has tried to supply housing, but the demand outweighs the supply, so housing eventually becomes crowded and unsanitary.
While the Aboriginal people now receive what they call "sit-down pay" (deposited into bank accounts for their use) this monetary support means the younger generation has lost the ability to live off the land; they have no "need" for work, nor do they see a role for education. As I found out, healthcare is also on the back burner. Clinics and transportation are available, but the people find it difficult to attend. Both preventive care and early detection are not yet embraced; unemployment is six times the rate of non-aborigines; and truancy in the primary schools is at 60%.
Cultural practices are also profoundly different; we do try to understand but find it difficult. This is demonstrated by the following story: I recently had a patient arrive with internal bleeding that occurred as a result of repetitively jumping out of a tree onto her abdomen. This was her way to demonstrate grief over the death of a grandparent and - coupled with an undiagnosed bleeding disorder - it almost cost her life. Also, once a person dies you can no longer view their image or say their name. This practice to honor family ties is just one instance of cultural differences.
Despite the aforementioned "doom and gloom", the Aborigine culture is slowly gaining ground - with the granting of full citizenship in 1967 and the return of their traditional ancestral land. If there is any area that has defined the renaissance of Australian Indigenous cultural globally, it is visual art. We have enjoyed learning about and collecting art from the different regions where we have worked. One of the most telling films of Aboriginal history is the film The Rabbit Proof Fence depicting a time when half-caste mixed blood children were taken from their homes and placed in missions in the hope of educating them and eventually "breeding out" the black color. It is a very powerful story.
On another note, the rainy season has arrived with its hot humid conditions and rain that comes in horrific downpours, but we're enjoying the adventure as always.
Happy New Year to All,
Kathy and Molly
Dr. Kathy Starkey, an OB/GYN, and her partner, Molly Evans, have chosen locum tenens as a permanent lifestyle (they're currently on their seventh assignment). Their adventures have taken them to New Zealand's North and South Islands, the Cayman Islands, Western Australia, the small Australian state of Tasmania and now the Top End of Australia (in the Northern Territory). Read more of their adventures right here. And if you haven't subscribed to this blog yet - do it now!
Topics: Dr. Kathryn Starkey, Molly Evans, Cadel Evans, Top End of Australia, Katherine AU, The Rabbit Proof Fence, Australian Indigenous Aborigines


text:
from __future__ import unicode_literals

from django.db import models, migrations
from tests.django_app.models import CustomField
from django.conf import settings

class Migration(migrations.Migration):

    dependencies = [
        ('auth', '0001_initial'),
        ('contenttypes', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='Client',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('username', models.CharField(max_length=20)),
                ('city', models.CharField(unique=True, max_length=20)),
                ('name', models.CharField(max_length=50)),
                ('created_at', models.DateField(auto_now_add=True)),
                ('updated_at', models.DateTimeField(auto_now=True)),
                ('score', models.IntegerField(default=50)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Customer',
            fields=[
                ('user_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to=settings.AUTH_USER_MODEL)),
                ('name', models.CharField(max_length=100)),
            ],
            options={
                'abstract': False,
                'verbose_name': 'user',
                'verbose_name_plural': 'users',
            },
            bases=('auth.user',),
        ),
        migrations.CreateModel(
            name='Door',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('size', models.PositiveIntegerField()),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Hat',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('color', models.CharField(max_length=50, choices=[(b'RD', b'red'), (b'GRN', b'green'), (b'BL', b'blue')])),
                ('brend', models.CharField(default=b'wood', max_length=10)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Hole',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('title', models.CharField(max_length=16)),
                ('size', models.SmallIntegerField()),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Message',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('content', models.TextField()),
                ('client', models.ForeignKey(to='django_app.Client')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Number',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='ColorNumber',
            fields=[
                ('number_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='django_app.Number')),
                ('color', models.CharField(max_length=20)),
            ],
            options={
            },
            bases=('django_app.number',),
        ),
        migrations.CreateModel(
            name='PointA',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='PointB',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Rabbit',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('title', models.CharField(max_length=16)),
                ('username', models.CharField(unique=True, max_length=16)),
                ('active', models.BooleanField(default=True)),
                ('email', models.EmailField(max_length=75)),
                ('text', models.TextField(max_length=512)),
                ('created_at', models.DateField()),
                ('updated_at', models.DateTimeField()),
                ('opened_at', models.TimeField()),
                ('percent', models.FloatField()),
                ('money', models.IntegerField()),
                ('ip', models.IPAddressField()),
                ('ip6', models.GenericIPAddressField(protocol='ipv6')),
                ('picture', models.FileField(upload_to=b'/tmp')),
                ('some_field', models.CommaSeparatedIntegerField(max_length=12)),
                ('funny', models.NullBooleanField()),
                ('slug', models.SlugField()),
                ('speed', models.DecimalField(max_digits=3, decimal_places=1)),
                ('url', models.URLField(default=b'', null=True, blank=True)),
                ('file_path', models.FilePathField()),
                ('object_id', models.PositiveIntegerField()),
                ('error_code', models.PositiveSmallIntegerField()),
                ('custom', CustomField(max_length=24)),
                ('content_type', models.ForeignKey(to='contenttypes.ContentType')),
                ('binary', models.BinaryField()),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Silk',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('color', models.CharField(max_length=20)),
                ('hat', models.ForeignKey(to='django_app.Hat')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Simple',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('value', models.IntegerField()),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Tag',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('title', models.CharField(max_length=20)),
                ('customer', models.ForeignKey(blank=True, to='django_app.Customer', null=True)),
                ('messages', models.ManyToManyField(to='django_app.Message', null=True, blank=True)),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='Through',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('pointas', models.ForeignKey(to='django_app.PointA')),
                ('pointbs', models.ForeignKey(to='django_app.PointB')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
        migrations.AddField(
            model_name='rabbit',
            name='one2one',
            field=models.OneToOneField(to='django_app.Simple'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='pointa',
            name='other',
            field=models.ManyToManyField(to='django_app.PointB', through='django_app.Through'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='number',
            name='doors',
            field=models.ManyToManyField(to='django_app.Door'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='number',
            name='wtf',
            field=models.ManyToManyField(related_name='wtf_rel_+', to='django_app.Number'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='hole',
            name='owner',
            field=models.ForeignKey(to='django_app.Rabbit'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='hat',
            name='owner',
            field=models.ForeignKey(blank=True, to='django_app.Rabbit', null=True),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='door',
            name='hole',
            field=models.ForeignKey(to='django_app.Hole'),
            preserve_default=True,
        ),
        migrations.AddField(
            model_name='door',
            name='owner',
            field=models.ForeignKey(blank=True, to='django_app.Rabbit', null=True),
            preserve_default=True,
        ),
    ]

redpajama_set_name: RedPajamaGithub


text:
HB 4545
House Public Education
Senate Education
Harold Dutton Jr.
Room 3N.5
Larry Taylor
Relating to the assessment of public school students, the establishment of a strong foundations grant program, and providing accelerated instruction for students who fail to achieve satisfactory performance on certain assessment instruments.
Estimated Two-year Net Impact to General Revenue Related Funds for HB4545, As Engrossed: a negative impact of ($146,860,972) through the biennium ending August 31, 2023. The bill would make no appropriation but could provide the legal basis for an appropriation of funds to implement the provisions of the bill.
HB 4545 would remove certain test-based promotional requirements and replace those with an Accelerated Learning Committee for students who perform unsatisfactorily on math and reading tests in third, fifth, and eight grade.
The bill would also establish the Strong Foundations Grant Program to "to implement a rigorous school approach that combines high-quality instruction, materials, and support structures" for pre-k through fifth grade.
Texas Action is neutral on HB 4545.


As a New Year special offer, we got a special on the NaturesLens Puffins of Fair-Isle Photography Holiday, we've taken £200 of the price – for a limited number of places the price is reduced to £999 from £1199 – the holiday package to photography the Shetland Puffins still includes a flight from Sumburgh to Fair Isle, several days on Fair Isle on a full-board basis, & a return ferry – dates for the trip are July 8th – 13th 2017.
The trip will be lead by Kevin Morgans & he will be on hand to offer photography guidance & to spot opportunities to ensure that you get the best from the the photography tour that you can.
You don't have to pay the full amount today, a deposit of £200 holds your place on the Shetland Puffins of Fair-Isle, with the balance due in early April 2017.
View the full details of the Shetland Puffins photography trip here.
This entry was posted in announcements, photography holidays, scotland, workshops and tagged atlantic puffin, fair isle, puffin, shetland, special offer.
Whatever your query regarding our photography holiday, £200 off the Shetland Puffins of Fair-Isle photography tour, please fill out your details below & we will get back to you.


text:
Gilles Guérin (født 1611 i Paris, død 26. februar 1678 sammesteds) var en fransk billedhugger.

Han udførte mange dekorative arbejder for Louvre og Versailles, for privatbygninger (Chateau de Maisons) og kirker og vandt især ry ved sine reliefportrætter (René Descartes i Saint-Étienne-du-Mont). Han blev medlem af Akademiet (på Vierge og Atlas) og virkede til sin død som professor ved Akademiet i Paris. To af hans mest ansete gravmonumenter, over hertugen af Vieuville (1652) og hans hustru (1663), kom til Louvre.

Kilder

Billedhuggere fra Frankrig
Professorer fra Frankrig
Franskmænd i 1600-tallet
Personer fra Paris
Medlemmer af Académie royale de peinture et de sculpture


Which Retirement Pay Plan are you under?
The Department of Veterans Affairs (VA), Veterans Services Outreach web site is designed to assist surviving spouses and dependents of veterans. You can find information on benefits and assistance for eligible survivors and dependents of veterans who honorably served in our Nation's armed forces.
The TSP is a retirement savings plan for civilians who are employed by the United States Government and members of the uniformed services. You can find information here about contribution limits, IRS Elective Deferral Limits, Loan Interest Rate Index, Annuity Interest Rate Index.