text:
\section{Introduction}

Since about 15 years there exists a wave of interdisciplinary research activities under the topic
''neural networks''. Neurobiologists, computer scientists, mathematicians, physicists,
psychologists, and linguists are making a more or less common effort to understand the
cooperative properties of a system of interacting neurons [Hertz et al 1991]. Meanwhile, the
initial excitement
and exaggerated promises have been replaced by practical research programs, but much has
been achieved and many interesting and unexpected results have been obtained.

The research on neural networks may be classified into three objectives:
\begin{enumerate}
\item Neurobiology: The material basis of our brain are about $10^{11}$ neurons, each of
which is directly connected to about $10^3$ other ones via synapses. We know a lot about
these single units and their anatomical and functional organization. However, we are still far
away from understanding learning, association, memory, recognition and generalization on the
basis of interacting neurons and their synaptic plasticity. It may be a philosophical problem
whether mind, soul, creativity and consciousness can be understood by collective
properties of a system of nerve cells. But there is a good chance to elucidate the basic
properties of a real neural network by simple models.

\item Computer science: There exists a variety of algorithms which use concepts from real
neural networks. Simple units represent information and interact by synaptic weights. Such
systems are trained by a set of examples. After the training phase, in which  the synaptic
weights are adapted to the presented examples, the network is able to achieve a knowledge
about the rule which produced the examples; it can generalize. These algorithms are called
neural networks or neurocomputer; they are presently applied to a large variety of problems in
engineering, science and economy. They have several advantages compared to standard
approaches, and there is  hope to solve problems by neural networks which are too hard for
methods of rule and data based algorithms of artificial intelligence.

\item Physics: Neural networks definitely belong to the class of complex systems, which are
characterized  by nonlinear dynamics,  feedback and  macroscopic properties emerging
from a huge number of interacting units. In general, physics is interested in understanding such
systems in terms of mathematical relations, scaling laws, phase transitions etc.
\end{enumerate}

In physics mathematical modelling of nature has been very successful. However, it is not clear
at all whether such a complex system like the brain can be described by a mathematical
language, by simple relations between macroscopic functions and microscopic mechanisms.

On the other side, the full quantum mechanical description of an iron solid is not possible,
either. Nevertheless one gains a lot of insight into the spontaneous magnetic ordering below a
critical temperature if one studies the Ising model, which replaces the rather complex iron atom
by a simple binary unit interacting with its neighbors. With this analogy it is definitely useful to
investigate simple units, which model a few essential mechanisms of neurons and synapses, and
to  study the cooperative behaviour of such interacting units. It is not obvious at all, whether
such a system can store an infinite number of patterns with one set of synapses, learn from
examples and generalize. Many questions can only be answered from a mathematical
calculation.

In this talk I want to emphasize the contribution of statistical
physics to the theory of neural computation. Using models and methods
from the physics of condensed matter one has been able to calculate
the properties of neural networks. This research program uses methods
developed already at the beginning of this century by L.~Boltzmann and
J.~W.~Gibbs. In 1975 S.~F.~Edwards and P.~W.~Anderson, S.~Kirkpatrick
and D.~Sherrington developed a theory of spin glasses [Fischer and
Hertz 1991] which was extended to neural networks by J.~J.~Hopfield
[1982]. The first analytic solution of attractor networks succeeded in
1985 [Amit et al 1987]. The statistical mechanics of learning from
examples was pioneered by the late E.~Gardner [1988]. These approaches
opened a new field of research, which produced a lot of interesting
results [for reviews see Watkin et al 1993, Opper and Kinzel 1996].
In view of the big challenge to understand the brain, the statistical
physics of neural networks will definitely survive over the next
century.

Long before the approach of statistical mechanics, mathematical models of neural networks
have been investigated in detail with great success [Hertz et al 1991]. But in my talk I want to
demonstrate
on a few examples, to what extend the physics approach is able to ask questions and obtain
results which are different from the approach of mathematics and computer science. In
particular I think that only the methods of physics can calculate discontinuous and singular
properties of infinitely large networks. Hence, in this paper I discuss examples from attractor
networks, generalization, structure recognition,
Bayesian estimate, on-line training, noise estimation and time series generation, which show
discontinuous behavior as a function of model parameters or the size of the training set.

This talk is not supposed to be a review. I apologize to all of my colleagues whose important
contributions to the theory of neural networks are not mentioned.

\section{Perceptron}

The simplest model of a neural network has already been introduced by Rosenblatt in 1960. It
consists of an input layer of ''neurons" $S_i\; , i = 1, ..., N$, which take only binary
values $S_i \in \{-1, +1\}$. The activity $S_i$ of each neuron is transmitted by ''synapses"
$W_i \in \real $ to an output neuron $\sigma$ as shown in Fig.~1. The output reacts to the
sign of the ''postsynaptic potential",
\begin{equation}\label{eins}
\sigma = \mbox{sign} \sum\limits^{N}_{i=1} W_i S_i =  \mbox{sign}\; \boldsymbol{W}
\cdot
\boldsymbol{S}
\end{equation}

\begin{figure}
\begin{center}
\setlength{\unitlength}{1pt}
\begin{picture}(170,130)(0,0)
\put(0,0){\makebox(0,0)
{\special{psfile=perc.ps angle=-90 hoffset=-180
          voffset=180 hscale=38 vscale=38}}}
\put(0,0){\makebox(0,0)
{\special{psfile=multilayer.ps angle=-90 hoffset=22
          voffset=182 hscale=40 vscale=40}}}
\put(-115.00,130.00){\makebox(0,0)[lc]{$S_1$}}
\put(-85.00,130.00){\makebox(0,0)[lc]{$S_2$}}
\put(45.00,130.00){\makebox(0,0)[lc]{$S_N$}}
\put(102.00,130.00){\makebox(0,0)[lc]{$S_1$}}
\put(252.00,130.00){\makebox(0,0)[lc]{$S_N$}}

\put(-100.00,75.00){\makebox(0,0)[lc]{$w_1$}}
\put(-5.00,75.00){\makebox(0,0)[lc]{$w_N$}}

\put(105.00,85.00){\makebox(0,0)[lc]{$\wvec_1$}}
\put(248.00,85.00){\makebox(0,0)[lc]{$\wvec_K$}}
\put(132.00,40.00){\makebox(0,0)[lc]{$1$}}
\put(212.00,40.00){\makebox(0,0)[lc]{$1$}}

\put(-10.00,15.00){\makebox(0,0)[lc]{$\studout=g\p{\wvec\cdot\Svec}$}}
\put(191.00,5.00){\makebox(0,0)[lc]{$\studout=f\p{\sum_{i=1}^K g\p{\wvec_i\cdot\Svec}}$}}
\end{picture}
\end{center}
\caption{Architecture of the perceptron (left) and the committee machine(right). }
\end{figure}

In the training phase this network, which was called ''perceptron", receives a set of training
examples $(\sigma^{\nu}, \boldsymbol{S}^{\nu}), \; \nu = 1, ... ,\alpha N$. It changes its
weight
$W_i$ such that a maximum number of examples is correctly mapped by Eq.~(\ref{eins}). A
simple algorithm has been investigated by Rosenblatt (see Hertz et al 1991): It presents the
examples in
an arbitrary sequence. If an example is not correctly classified, that is if $\boldsymbol{W} (t) \cdot
\boldsymbol{S}^{\nu} \sigma^{\nu} < 0$, then
\begin{equation}\label{zwei}
\boldsymbol{W}(t+1) = \boldsymbol{W}(t) + \frac{1}{N}\; \boldsymbol{S}^{\nu} \cdot
\sigma^{\nu}
\end{equation}
There exists a convergence proof for this algorithm: If the examples can be mapped correctly
by any perceptron, Eq.~(\ref{eins}) with weights $\boldsymbol{W}^*$, then the perceptron
rule
Eq.~(\ref{zwei}) finds a solution, i.~e.~the algorithm stops.

The Rosenblatt training rule stems from neurobiology, as proposed by D.~Hebb in 1949:
Each synapse reacts to the neuronal activities at its two ends. Here we need an additional
influence of the postsynaptic potential.

The perceptron implements a linear separable Boolean function, which
has an interesting geometrical interpretation: $\boldsymbol{W} \cdot
\boldsymbol{S} = 0$ defines a hyperplane in an $N$--dimensional space
of inputs $\boldsymbol{S}$, the weight vector $\boldsymbol{W}$ is
normal to this plane. On the side of the vector $\boldsymbol{W}$ the
perceptron classifies each input $\boldsymbol{S}$ to $\sigma = + 1$
(black, correct, ...), on the other side the label is $\sigma =
-1$ (white, wrong, ...) see Fig.~2.\\
\begin{figure}
\centerline{
\epsfysize=5cm
\epsffile{fig2.ps}
}
\caption{  Space of input vectors $\boldsymbol{S}$. The weight vector
$\boldsymbol{W}$ of the perceptron defines a hyperplane in the $N$--dimensional space,
which separates the labels $\sigma$ of the input vectors $\boldsymbol{S}$. In the shaded
region the labels of teacher $\boldsymbol{B}$ and student perceptron $\boldsymbol{W}$ are
different. }
\end{figure}

\noindent Now we consider a set of  $\alpha N$ many points $\boldsymbol{S}^{\nu}$ in $N$
dimensions. How many sets of labels $\{\sigma^{\nu}\}$ can be represented by any perceptron?
Surprisingly this problem which is important for the theory of neural computation was already
solved by the Swiss mathematician Ludwig Schl\"afli in the last century [Schl\"afli 1852]. If
any subset of
$N$ points is linearly independent, then the number $C$ of possible sets of labels
$\{\sigma^{\nu}\}$ is given by
\begin{equation}\label{drei}
C = 2 \sum\limits^{N-1}_{i=0}  {\alpha N-1 \choose i} .
\end{equation}
For $\alpha <1$ all labels can be produced by a perceptron, i.~e.~
$C=2^{\alpha N}$. For $\alpha < 2$ there is a large fraction of labels
which can be separated by a hyperplane.  For $\alpha >2$ only a tiny
fraction of patterns can be stored, this fraction disappears for $N
\rightarrow \infty$.  This result has consequences for the associative
memory which will be discussed in the following section: In the limit
of $N \rightarrow \infty$ a network with $N$ neurons can store up to
$2N$ random patterns.
\begin{figure}
\centerline{
\epsfysize=5cm
\epsffile{fig3.eps}
}
\caption{ Two dimensional projections of 250 points in 200 dimensions. The points are labelled
randomly. The perceptron algorithm finds a hyperplane which separates different labels. }
\end{figure}

The geometry of this result is shown in Fig.~3. 250 points are located in a 200 dimensional
space and randomly colored black or white with equal probability. Now we are
moving in the space of points and would like to find a view where black and white is clearly
separated by a single gap. From Eq.~(\ref{drei}) we find with $N=200$ and $\alpha N=250:
\hspace{0.5cm} C/2^{250} \simeq 1-4\cdot 10^{-23}$; that means for random labels it is
almost sure that one can find such a view. In fact the Rosenblatt algorithm, Eq.~(\ref{zwei}),
found the solution shown in Fig.~3.

\section{Attractor networks}
The perceptron  is the ''atom" of all neural networks. Many of such elementary units can be
composed to a large and complex network. Here we consider an attractor network which
consists of $N$ neurons $S_i$ as before. But now each element $S_i$ is connected to any other
element $S_j$ by a coupling $W_{ij} \in \real$, as illustrated in Fig.~4.
\begin{figure}
\centerline{
\epsfysize=5cm
\epsffile{fig4.ps}
}
\caption{ A set of six perceptrons is connected to form an attractor network. }
\end{figure}

We want to store $\alpha N$ many patterns $S^{\nu}_{i} \in \{-1, +1\}\;  ; i=1, .., N\; ; \nu =1,
..., \alpha N$. If we use the Rosenblatt rule, Eq.~(\ref{zwei}), without the additional
condition, we obtain the Hebbian couplings
\begin{equation}\label{vier}
W_{ij} = \frac{1}{N} \sum_{\nu} S^{\nu}_{i} S^{\nu}_{j} \,\,\,\,\,\,\,\, (i \neq j).
\end{equation}
Since each input $S_j$ is output of a perceptron with weights $W_{jk}$, we can define a
dynamics of the configuration of neurons, $\boldsymbol{S}$. For instance, for each neuron
$S_i$ we consider the local field
\begin{equation}\label{fuenf}
h_i = \sum_j W_{ij} \; S_j (t)
\end{equation}
where $t$ is a discrete time index. Now we define a stochastic dynamics by the probability $P$
to find neuron $S_i$ in the state $S \in \{+1, -1\}$ in the next time step $t+1$:
\begin{equation}\label{sechs}
P[S_i (t+1) =S] = \frac{e^{\beta h_{i}S}}{2 \cosh (\beta h_{i}S)}
\end{equation}
$\beta$ is a parameter which measures the noise level of the dynamics. For $\beta \rightarrow
\infty$ we obtain the noiseless deterministic equation
\begin{equation}\label{sieben}
S_i (t+1) = \mbox{sign} \sum_j W_{ij} S_j (t).
\end{equation}
This model was introduced by Hopfield [1982]. He noticed that the dynamics of the neurons is
nothing else than the usual Monte Carlo procedure to obtain thermal equilibrium. Since the
couplings are symmetric, $W_{ij}= W_{ji}$, the stationary state is given by a Boltzmann
distribution
\begin{equation}\label{acht}
P(\boldsymbol{S}) = \exp (-\beta H(\boldsymbol{S})) / Z
\end{equation}
with a Hamiltonian
\begin{equation}\label{neun}
H(\boldsymbol{S}) = -\frac{1}{2} \sum_{i \neq j} W_{ij} S_{i}S_{j}
\end{equation}
This is the main advantage of equlibrium statistical mechanics: The dynamics
$\boldsymbol{S} (t)$ is replaced by a summation over all possible states $\boldsymbol{S}$.
Instead of solving a system of $N$ strongly coupled nonlinear equations, one has to
calculate the partition sum
\begin{equation}\label{zehn}
Z = \sum_{\{\boldsymbol{S}\}} \exp \left[-\frac{\beta}{2} \sum_{i\neq j} W_{ij} S_i S_j \right].
\end{equation}
In the thermodynamic limit of infinitely many neurons, $N \rightarrow \infty$, and infinitely
many patterns, $\alpha = $const., the partition sum $Z$ was solved exactly by Amit et al
[1987] using
the replica method. There are two main steps in the calculation:
\begin{enumerate}
\item The free energy $f= - \ln Z/ \beta$ is averaged over all possible sets of patterns
$\{\boldsymbol{S}^{\nu}\}$. It can be shown that the average value gives the same results as
the value $f$ for a single, randomly chosen set of patterns. Hence, this calculation yields results
for a typical situation.
\item The sum over $2^N$ states in $\ln Z$ is performed for fixed order parameters. The
minimum of $f$ as a function of these quantities yields their physical values, which describe the
stationary state. Hence, the complex system of interacting neurons is described exactly by a
few order parameters.
\end{enumerate}

The first step is done using the replica method:
\begin{equation}\label{elf}
\langle \ln Z \rangle_{\{\boldsymbol{S}^{\nu}\}} = \lim_{n \rightarrow 0}
\frac{\partial}{\partial n}  \langle Z^n \rangle_{\{\boldsymbol{S}^{\nu}\}}.
\end{equation}
In the second step one is interested in the overlap between the state
$\boldsymbol{S}$ and one of the patterns $\boldsymbol{S}^{\nu}$. Let
us assume that the first pattern $\boldsymbol{S}^1$ has the form of an
''{\bf A}" as shown in Fig.~4 with $N=400$. The other 31 patterns
consist of random bits. If the initial state $\boldsymbol{S}(0)$ has
an overlap to the first pattern, for instance if it is the noisy
''{\bf A}" of Fig.~5, then after a few steps given by
Eq.~(\ref{sieben}) the network relaxes to the stored information more
or less completely.
\begin{figure}
\centerline{
\epsfysize=4cm
\epsffile{fig5.eps}
}
\caption{ An initial state of an attractor network relaxes to one of the stored patterns.
(From Kinzel 1985)}
\end{figure}

The statistical mechanics gives information about the possible final states of the dynamics.
Here we are interested in the overlap after the relaxation:
\begin{equation}\label{zwoelf}
m_A = \frac{1}{N} \boldsymbol{S}\; \boldsymbol{S}^A.
\end{equation}
It turns out that one obtains this order parameter if one calculates the free energy. The
overlaps
\[
m_{\nu} = \frac{1}{N} \; \boldsymbol{S} \cdot  \boldsymbol{S}^{\nu}
\]
to the other 31 patterns are of the order of $1/ \sqrt{N}$. However, their sum $r$ is an
additional order parameter
\[
r = \frac{1}{\alpha} \sum\limits^{\alpha N}_{\nu =2} m_{\nu}^2
\]
$r$ measures the fluctuation of the final state to the rest of the patterns. Finally there is an
order parameter $q$ which measures the complexity of the space of possible solutions
$\boldsymbol{S}$. Like in the theory of spin glasses, it signals an additional order of the
stationary states which has no simple interpretation [Fischer and Hertz 1991].

The theory of attractor networks has close similarities to the theory of an Ising ferromagnet
with infinite range interactions. In both cases energy and entropy can be expressed in terms of
order parameters. For the ferromagnet one obtains an implicit equation for the spontaneous
magnetic order $m$ [Yeomans 1992]
\begin{equation}\label{dreizehn}
m = \tanh \beta Jm \, \, .
\end{equation}
For the attractor network one finds
\begin{equation}\label{vierzehn}
m_A = \langle \tanh (\beta m_A + \beta \sqrt{\alpha r}z )\rangle_z
\end{equation}
where the average is performed over a Gaussian distributed quantity $z$. There are additional
equations for $r$ and $q$. Hence, compared to the ferromagnet, the $\alpha N-1$ patterns add
a noise term to the local fields.
\begin{figure}
\centerline{
\epsfysize=6cm
\epsffile{fig6.ps}
}
\caption{ Schematic phase diagram of the Hopfield model. (From Amit et al, 1987) }
\end{figure}

Fig.~6 shows the result of the analytic calculation [Amit et al 1987]. In the
noise--load plane one obtains several phases, which are well separated
in the thermodynamic limit. For strong noise, $T = 1/\beta >1$, or for
high load, $\alpha > 0.14$, the network cannot recognize the stored
patterns at all. Nevertheless there is a spin glass order for low
noise $T < T_{g}$ with $q>0$. Only for $T <T_m (\alpha)$ the network
can relax to final states which have an overlap to one of the stored
patterns. This overlap jumps discontinuously to zero at $T_m$. For $T
<T_c (\alpha)$ this retrieval state has the lowest free energy,
i.~e.~it is thermodynamically stable. Note that also for the
deterministic dynamics $(T=0)$ there is a discontinuous drop to zero
retrieval at $\alpha = \alpha_c \simeq 0.14$. For $\alpha < \alpha_c$
the network restores stored information very well, for $\alpha >
\alpha_c$ the network relaxes to final states which have nothing in
common with the stored patterns.

According to Schl\"afli there are couplings with a storage capacity
of $\alpha_c = 2$, but these interactions are not symmetric and one
cannot apply statistical mechanics to solve the corresponding
attractor network. A network with $\alpha_c = 1$ has been analysed by
Kanter and Sompolinsky [1987].

In summary, the attractor network functions as an associative memory. It is a distributed
memory, since all patterns are stored in all couplings. It is content addressable, since a state
with a partial information relaxes to the complete one. Even with a stochastic dynamics it
performs well, if the noise level $T$ and storage capacity $\alpha$ are not too high. There is a
sharp, discontinuous transition between good and zero performance. Processing of information emerges
as a cooperative effect from a large number of simple interacting units.

\section{Generalization}
We have seen how a network of mutually interacting perceptrons can work as an
associative memory. But already the simple perceptron itself has interesting properties. It can
learn from examples and recognize an unknown rule.

Consider a perceptron with a weight vector $\boldsymbol{W}$, as in
Eq.~(\ref{eins}). We will call this perceptron the ''student". It
obtains a set of examples $(\sigma_{B}^{\nu}, \boldsymbol{S}^{\nu} ),
\nu = 1, ..., \alpha N$ from a ''teacher". In the simplest case the
teacher is another perceptron with a weight vector $\boldsymbol{B}$.
To what extent can the student gain information about the vector
$\boldsymbol{B}$ if the only available information is the set of
$\alpha N$ many examples? The patterns $\boldsymbol{S}^{\nu}$ are
selected randomly and $\sigma^{\nu}_{B}$ is the output of the teacher,
\begin{equation}\label{fuenfzehn}
\sigma^{\nu}_{B} = \mbox{sign} \; \boldsymbol{B} \cdot \boldsymbol{S}^{\nu}
\end{equation}
As before we are interested in the limit $N \rightarrow \infty$ and $\alpha =$constant. \\
We have to consider two processes:
\begin{enumerate}
\item The training phase:\\
The student network is trained by use of the examples, it tries to decrease the training error
\begin{equation}\label{sechzehn}
\varepsilon_t (\boldsymbol{W}) = \sum\limits^{\alpha N}_{\nu =1} \;\; \Theta [-
\sigma^{\nu}
\cdot \sigma^{\nu}_{B} ]
\end{equation}
$\Theta(x)$ is the step function, it is zero if the student  reproduces the example
$\boldsymbol{S}^{\nu}$ correctly.

\item  The test phase:\\
Now the student receives an input $\boldsymbol{S}$ which  has not been presented before. It gives
the
answer $\sigma = \mbox{sign}\;  \boldsymbol{W} \cdot \boldsymbol{S}$, which may be
different from the
answer by the teacher, $\sigma_B = \mbox{sign}\; \boldsymbol{B} \cdot \boldsymbol{S}$. The
probability
of disagreement or the generalization error is defined by an average over all possible input
vectors $\boldsymbol{S}$:
\begin{equation}\label{siebzehn}
\varepsilon_g = \langle \Theta [- \sigma \; \sigma_B] \rangle_{\boldsymbol{S}}
\end{equation}
From Fig.~2 one can see that $\varepsilon_g$ is determined by the angle between the
weight vectors of the student and the teacher perceptron
\begin{equation}\label{achtzehn}
\varepsilon_g = \frac{1}{\pi} \arccos \frac{\boldsymbol{B} \cdot
\boldsymbol{W}}{|\boldsymbol{B}| \, |\boldsymbol{W}|}
\end{equation}
\end{enumerate}
Training and generalization of the perceptron has been studied in
detail using methods of statistical mechanics [see e.g. Watkin et al
1993, Opper and Kinzel 1996] on a simple scenario where the weights
are restricted to binary values,
$W_i \in \{+1, -1\}$ and $B_i \in \{+1, -1\}$. The student perceptron is
trained by a stochastic algorithm, for instance by a Monte Carlo
procedure similar to Eq.~(\ref{sechs}). But now we have a stochastic
dynamics of the synaptic weights $W_i$ instead of the neurons $S_i$,
which leads to a thermal equilibrium given by
\begin{equation}\label{neunzehn}
P(\boldsymbol{W}) = \exp [-\beta \; \varepsilon_t (\boldsymbol{W})]  /Z
\end{equation}
As before we do not have to solve the complex nonlinear dynamics of the weights
$\boldsymbol{W}(t)$ but rather calculate the partition sum
\begin{equation}\label{zwanzig}
Z = \sum\limits_{\{\boldsymbol{W}\}} \;\; \exp [-\beta \; \varepsilon_t (\boldsymbol{W})]
\end{equation}
Again we have to perform two steps:
\begin{enumerate}
\item Average $\ln Z$ over all possible sets of examples $\{\boldsymbol{S}^{\nu}\}$.
\item Evaluate the sum of $2^N$ states $\boldsymbol{W}$ by introducing order parameters.
\end{enumerate}
In the limit of large noise, $\beta \rightarrow 0$, the calculation turns out to be very easy
[Seung et al 1992]:
The only order parameter is the overlap $R$ between student and teacher,
\begin{equation}\label{einundzwanzig}
R = \frac{1}{N} \; \boldsymbol{B} \cdot \boldsymbol{W}
\end{equation}
The training error of Eq.~(\ref{zwanzig}) can be replaced by the generalization error
\begin{equation}\label{zweiundzwanzig}
\alpha \; \varepsilon_g = \frac{\alpha}{\pi} \arccos R
\end{equation}
and the entropy is the well known mixture entropy of binary variables
\begin{equation}\label{dreiundzwanzig}
S(R) = \frac{1}{2} [ (1+R) \; \ln(1+R) + (1-R)\; \ln(1-R)] + \ln2
\end{equation}
$R$ is determined by the minimum of the free energy
\begin{equation}\label{dreiundzwanziga}
f(R) = \alpha \varepsilon_g ( R ) - T S( R)
\end{equation}
Note that the product $\beta \alpha$ in the limit $\beta \rightarrow 0$ is the only parameter of
the model, hence a large noise has to be compensated by a large number of examples.
\begin{figure}
\centerline{
\epsfysize=5cm
\epsffile{fig7.ps}
}
\caption{ Generalization error $\varepsilon_g$ as a function of the size $\alpha$ of the
training set (schematic). The binary perceptron is trained stochastically for large noise $(\beta
\rightarrow 0)$. The dotted line describes the first order phase transition to perfect
generalization in thermal equilibrium. The solid line extends to a metastable state. (From Seung
et al 1992) }
\end{figure}

One minimum of $f$ is always $R=1$, i.~e.~, the student perfectly recognizes the teacher.
However, for $\beta\alpha < 2.08$, the system has an additional minimum at $R<1$ which is
the global one for $\beta\alpha < 1.69$. Fig.~7 shows the generalization error as a function
of the fraction of learned patterns. There is a discontinuous transition from poor to perfect
generalization, similar to a first order phase transition in solid state physics. Both of the
transitions are characterized by metastable states and hysteresis loops. This process of
sudden recognition appears even for a noisy training algorithm. A replica calculation shows
that the transition qualitatively extends to zero noise $T=0$. [Seung et al, 1992]

\section{Multilayer networks}
We have already seen how an attractor network can be built from many perceptrons.
Another interesting system which can be constructed from many elementary units is a
multilayer network, shown in Fig.~1. It consists of several layers of synaptic weights which
map the information coded in the neurons from top to bottom. It is important that such
networks can realize any function, if the number of hidden units (neurons in the interior layers)
is large enough.

The simplest multilayer network is a committee machine. It consists of $N$ input units, $K$
hidden units, $K$ weight vectors $\boldsymbol{W}_i, i=1, ..., K$ and one output unit
$\sigma$.
The weights of the second layer have the value $+1$, that means, that the output bit $\sigma$
is given by the majority of the $K$ perceptrons (= members of the committee) in the first layer,
\begin{equation}\label{vierundzwanzig}
\sigma = \mbox{sign} \left[ \sum\limits^K_{i=1} \;  \mbox{sign} \; \boldsymbol{W}_i \;
\boldsymbol{S} \right]
\end{equation}
This network is trained from a set of examples $(\sigma^{\nu}_B, \boldsymbol{S}^{\nu}), \nu
=
1, ..., \alpha N$. Note that the opinion of the majority is trained, not the opinion of each
member of the committee!

Here we consider the case, where the student is a committee machine with $K=3$ members.
The teacher is a simple perceptron with single weight vector $\boldsymbol{B}$. All the weights
are
assumed to be binary, $W_{ik}, B_i \, \in \, \{+1, -1\}$. To what extent can a complex
network gain information about a simple rule from a set of examples?

In analogy to the previous section we consider a stochastic training
algorithm. The training error $\varepsilon_t$ is the ''energy" of the
Gibbsian probability which describes the stationary state of the
stochastic algorithm. In the limit of high noise there are several
order parameters, which determine the energy, entropy and
generalization error. Firstly, there are the overlaps between the
members of the committee and the teacher,
\begin{equation}\label{fuenfundzwanzig}
R_i = \frac{1}{N} \boldsymbol{W}_i \cdot \boldsymbol{B} \;\;\; (i = 1, 2,3)\;\;.
\end{equation}
Secondly, the weight vectors of the committee have a mutual overlap
\begin{equation}\label{sechsundzwanzig}
Q_{ij} = \frac{1}{N} \boldsymbol{W}_i \boldsymbol{W}_j
\end{equation}
Their physical values are determined by minimizing the corresponding free energy.\\
Fig.~8 shows the result of the analytic calculation [Schwarze et al
1992]. For a small size of the training set ($\beta\alpha$ small) the
members of the committee react symmetrically, $R_1 = R_2 = R_3 < 1$
and the generalization error decreases continuously with $\alpha$.
By increasing the size of the training set, suddenly one of the members recognizes
the teacher perfectly, $R_1 =1, R_2 =R_3 <1$, and the error jumps to a
low value. Further increase of  $\beta\alpha$ leads to another
discontinuous transition to perfect recognition of the majority. Since
the majority vote is already determined by two members, the highest
entropy is achieved for $R_1 = R_2 =1$ and $R_3 = 0$.
\begin{figure}
\centerline{
\epsfysize=4cm
\epsffile{fig8.ps}
}
\caption{ Generalization error $\varepsilon_g$ as a function of the size of the training set
(schematic). A committee machine with binary weights and three hidden units is trained to a set
of examples given by a binary perceptron. (From Schwarze et al 1992) }
\end{figure}

Here again the
competition between energy and entropy leads to an interesting
discontinuous behavior of the generalization ability. Such sharp
transitions, which occur for infinitely large networks, only $(N
\rightarrow \infty)$, are not obvious. One needs the tools of
statistical mechanics to find and describe them.

\section{Parity machine}

Now we want to discuss another multilayer network, the parity machine with a tree
architecture shown in Fig.~9. It consists of $N$ input and $K$ hidden units. The input units
are grouped into
$K$ parts with $N/K$ neurons each. Each part is input of a perceptron with weights
$\boldsymbol{W}_i \; (i=1,...,K)$. The output of the whole network is given by the parity of
the outputs of the $K$ perceptrons,
\begin{equation}\label{siebenundzwanzig}
\sigma = \prod\limits^{K}_{j=1} \; \mbox{sign} \; (\boldsymbol{W}_j \cdot \boldsymbol{S})
\end{equation}
\begin{figure}
\centerline{
\epsfysize=3cm
\epsffile{fig9.ps}
}
\caption{ Parity machine with a tree architecture. Each of the three weight vectors
$\boldsymbol{W}_i$ is connected to  only one third of the input vector. The output bit is the product of
the three hidden units. }
\end{figure}

We consider the case, where both of the student and teacher networks are a parity machine
with the same number of units. The teacher network is presenting a set of examples given by
\begin{equation}\label{achtundzwanzig}
\sigma^{\nu}_{B} = \prod\limits^{K}_{j=1} \; \mbox{sign} (\boldsymbol{B}_j \cdot
\boldsymbol{S}^{\nu}) \; \; (\nu = 1, ..., \alpha N)
\end{equation}
The examples should be learned without errors. In this case we are interested in the volume
$V$ of all student vectors $\{ \boldsymbol{W}_1, ..., \boldsymbol{W}_K \}$ which learn the
set of examples $\{ (\sigma^{\nu}_{B}, \boldsymbol{S}^{\nu}) \}$ perfectly. $V$ is an
integral over a $N$--dimensional space and corresponds to the partition sum $Z$ of the
previous sections. The method of the calculation is similar as before:
\begin{enumerate}
\item Average $V$ over all possible sets of inputs $\{\boldsymbol{S}^{\nu}\}$ using the replica
method.
\item Calculate the integrals by introducing order parameters $R$ and $Q$, similar to Eqs.
(\ref{fuenfundzwanzig}) and (\ref{sechsundzwanzig}).
\end{enumerate}
In general, an additional average of $V$ over all possible teacher vectors
$\boldsymbol{B}_i$ is to be performed.

\begin{figure}
\centerline{
\epsfysize=5cm
\epsffile{fig10.eps}
}
\caption{ Generalization error as a function of the size of the training set for the parity
machine, which learns perfectly a set of examples given by a teacher parity machine. From left to
right: $K=1,2,3$ and $4$. (From Opper 1994) }
\end{figure}

The generalization error is shown as a function of the number of
learned examples in Fig.~10 [Opper 1994]. It reveals unexpected
properties of the network: For a large fraction of examples, $ 0 <
\alpha < \alpha_c (K)$, the network cannot generalize at all
$(\varepsilon_g = 1/2)$, although it stores of the order of $N$
examples perfectly! Zero training error does not imply an overlap
between student and teacher network, even for $\alpha >0$.

If the number of examples is increased to a critical threshold $\alpha_c (K)$, then the student
suddenly recognizes the rule, $\varepsilon_g$ jumps to a low value and decreases
asymptotically as $1/ \alpha$, independently of the number $K$ of hidden units. This property is
another surprise: The asymptotic behavior is not determined by the Vapnik--Chervonenkis
dimension, which diverges as $\ln K$ [Barkai et al 1990].

\section{Structure recognition}

Up to now we have discussed supervised learning, that means the input patterns $\boldsymbol{S}^{\nu}$
have the labels $\sigma^{\nu}$. But
there are many applications of neural networks where the labels are not given. In these cases
the task is to detect a structure in the data [Hertz et al 1991].
\begin{figure}[h]
\centerline{
\begin{turn}{-90}
\epsfysize=8cm
\epsffile{fig11.eps}
\end{turn}
}
\caption{ Projection of a distribution of data points. In one direction $\boldsymbol{B}$
  the overlap has a double peak distribution (b), in all orthogonal
  directions it is Gaussian distributed (a).  (From Biehl, 1997) }
\end{figure}

Consider for example two clouds of $\alpha N$ data points as shown in
Fig.~11. The $\boldsymbol{S}^{\nu}$ are distributed in a
$N$--dimensional space according to a mixture of two Gaussian
distributions [Biehl and Mietzner 1993]. This means, that there is a
direction $\boldsymbol{B}$ in data space where the projections
$y_{\nu} = \boldsymbol{B} \cdot \boldsymbol{S}^{\nu} / \sqrt{N}$ have
a double peak distribution. In any  direction $\boldsymbol{W}$
orthogonal to $\boldsymbol{B}$ the corresponding projections $x_{\nu}$
are Gaussian distributed with a single peak, as shown in Fig.~11. Note
that the lengths of all vectors $\boldsymbol{S}^{\nu}, \boldsymbol{W}$
and $\boldsymbol{B}$ are of the order of $N$, while the overlap
$\boldsymbol{S}^{\nu} \cdot \boldsymbol{B}$ is of the order of
$\sqrt{N}$.

Given the $\alpha N$ many data points, we want to find the direction
$\boldsymbol{B}$.  There exists a method, well known in engineering,
which is called ''Principal component analysis" and determines the
direction of maximal variance in data space [Hertz et al 1991]. In
fact there is an algorithm for neural networks which finds this
direction [Oja 1982]. For our example this means, that we want to find a
direction $\boldsymbol{W}$ which minimizes
\begin{equation}\label{neunundzwanzig}
H(\boldsymbol{W}) = - \sum\limits^{\alpha N}_{\nu = 1} (\boldsymbol{W} \cdot
\boldsymbol{S}^{\nu} )^2 / N
\end{equation}
The minimum of $H$ can be found by calculating the partition sum
\begin{equation}\label{dreissig}
Z = \int d^N \boldsymbol{W} \; \delta(\boldsymbol{W}^2 - N) \; \exp (- \beta H
(\boldsymbol{W}))
\end{equation}
in the limit of $\beta \rightarrow \infty$. Here we have again replaced the dynamics of the
algorithm by a summation over all possibilities. As before we have to average $\ln Z$ over all
possible data points $\boldsymbol{S}^{\nu}$. The evaluation of the $N$--dimensional integral
in the limit $N \rightarrow \infty$ yields the order parameter
\begin{equation}\label{einunddreissig}
R = \frac{1}{N} \boldsymbol{W} \cdot \boldsymbol{B}
\end{equation}
\begin{figure}[h]
\centerline{
\epsfysize=5cm
\epsffile{fig12.eps}
}
\caption{ Overlap $R$ between the direction $\boldsymbol{B}$ of Fig.~11 and the
weight vector $\boldsymbol{W}$ of the training algorithm. The theory (solid line) shows a
transition from zero to nonzero recognition with increasing number of data points in the limit
$N \rightarrow \infty$. The Monte Carlo simulation (points) for $N =1000$ show that finite size
effects smoothen the transition. (From Biehl and Mietzner, 1993) }
\end{figure}

The result of this calculation is shown in Fig.~12 [Biehl and Mietzner
1993]. Surprisingly one observes a sharp phase transition. For a small
number of data points the system cannot recognize the direction
$\boldsymbol{B}$ of separation at all. Above a critical number
$\alpha_c$ the symmetry $H(\boldsymbol{W}) = H(-\boldsymbol{W})$ is
spontaneously broken: $|R|$ increases with the deviation from
$\alpha_c$ similar to the magnetization in a ferromagnet.

Using the concepts of energy, partition sum and order parameter an unexpected sharp
transition from zero to good performance was found for the standard method of principal
component analysis.

\section{Bayesian estimate}

In the previous section we have found the structure of a data distribution by minimizing a cost
function. However, if one knows something about the structure of the data it is more efficient
to include this knowledge into the algorithm. Here we want to discuss this problem for the two
overlapping clouds of data considered in the previous paragraph, see Fig.~11.

The distribution of the data points $\boldsymbol{S}^{\nu}$ is given by
\begin{eqnarray}\label{zweiunddreissig}
P(\boldsymbol{S} | \boldsymbol{B} ; \rho) & \propto & \sum\limits_{\tau = \pm 1} \; \exp \left[
- \frac{1}{2} (\boldsymbol{S} - \frac{\rho \tau}{\sqrt{N}} \boldsymbol{B} )^2 \right]
\nonumber\\
& \propto & \exp [-\beta \; H(\boldsymbol{S}; \boldsymbol{B} , \rho) ]  \; \; (\beta = 1)
\end{eqnarray}
This distribution has two parameters: The vector $\boldsymbol{B}$ of length $N$ which gives
the direction of the cloud separation and the distance $\rho$ between the centers of the clouds.

Now let us assume we know the form of the distribution, Eq.(\ref{zweiunddreissig}), and want to
estimate its parameters $\boldsymbol{B}$ and $\rho$. Hence our model is for a given distance
$\tilde{\rho}$:
\begin{equation}\label{dreiunddreissig}
P (\boldsymbol{S} | \boldsymbol{W}) \propto \; \exp[- \beta H (\boldsymbol{S};
\boldsymbol{W}, \tilde{\rho})]
\end{equation}
Given the set of data points $\boldsymbol{S}^{\nu}, \nu = 1,..., \alpha N$, the a posteriori
distribution of directions $\boldsymbol{W}$ is given by the Bayes relation
\begin{equation}\label{vierunddreissig}
P(\boldsymbol{W} | \{\boldsymbol{S}^{\nu}\}) = \frac{1}{Z} \; \prod\limits^{\alpha N}_{\nu
=
1} \; \delta (\boldsymbol{W}^2 - N) \; \exp [- \beta H (\boldsymbol{S}^{\nu};
\boldsymbol{W}, \tilde{\rho}  )]
\end{equation}
There are several possibilities to estimate a direction $\boldsymbol{W}$ from this distribution
[Biehl 1997]: Their performance can be measured by the order parameter
\begin{equation}\label{fuenfunddreissig}
R = \frac{1}{N} \; \boldsymbol{W} \cdot \boldsymbol{B}
\end{equation}
which has a single value in the limit $N \rightarrow \infty$. For example one may maximize the
a posteriori distribution with respect to $\boldsymbol{W}$. The ''maximum likelihood"
corresponds to the minimum of
\begin{equation}\label{sechsunddreissig}
H (\boldsymbol{W}) = - \sum\limits^{\alpha N}_{\nu = 1} \; \ln \cosh
\frac{\tilde{\rho}}{\sqrt{N}} \; \boldsymbol{W} \cdot \boldsymbol{S}^{\nu}
\end{equation}
which can be studied by calculating $Z$ for $\beta \rightarrow \infty$, using the
replica method [Barkai and Sompolinsky 1994]. For small cloud separation $\tilde{\rho}$ the maximum
likelihood solution coincides with first principal component.
\begin{figure}[h]
\centerline{
\begin{turn}{-90}
\epsfysize=7cm
\epsffile{fig13.ps}
\end{turn}
}
\caption{ Phase diagram for Bayes estimation of the direction $\boldsymbol{B}$ of the
two clouds of data points in Fig.~11. $\alpha$ measures the number of data points and $\tilde
{\rho}$ is the estimated distance of the clouds relative to the true  $\rho = 1$. (From Biehl
1997) }
\end{figure}

Another possibility is to select a direction $\boldsymbol{W}$
according to the a posteriori distribution, Eq.
(\ref{vierunddreissig}), for instance by using the Monte Carlo method
[Watkin and Nadal 1994]. The result is obtained from calculating $Z$
for $\beta = 1$, again by averaging $\ln Z$ over the true distribution
of data points. The result of this Gibbs estimate is shown in Fig.~13 [Biehl 1997].
In the plane of $\alpha$, the size of the data set, and
$\tilde{\rho}$, the estimated distance between the clouds, there are
sharp phase transitions. Recognition $(R > 0)$ appears only for a
sufficiently large number of data points. It is better to use an
estimate $\tilde{\rho}$ which is larger than the true one $\rho = 1$,
since for small $\tilde{\rho}$ the critical fraction $\alpha_c$
diverges as $\tilde{\rho}^{-4}$.

\section{On--line training}

In the previous sections {\bf all} of the examples were presented in
the training phase of the neural network. The algorithm used the
training error with respect to all of the examples in order to find
the synaptic weights of the student network. For instance for the
Rosenblatt algorithm, Eq.~(\ref{zwei}) all  examples have to be
predented several times  before the algorithm stops.

Now we want to consider a different training algorithm. At each step
only one new example is presented. One does not have to store the
complete set of the examples, but the present weight vector
$\boldsymbol{W}$ is changed due to one new example $(\sigma^{\nu}_B ,
\boldsymbol{S}^{\nu})$. It turns out that such an ''on--line" training
is more efficient in terms of computational effort than the
''off--line" or ''batch" rules of the previous sections, if there are
enough examples available.

On--line learning leads to a stochastic differential equation for the
weight vector $\boldsymbol{W}(\nu)$, which becomes a deterministic one
for several order parameters in the limit $N \rightarrow \infty$ [Biehl
and Schwarze 1995, Saad and Solla 1995]. Usually, the dynamics of
on--line learning is not described by a Hamiltonian or a partition
sum, nevertheless there are discontinuous properties as a function of
model parameters.

Let us consider a two layer network with continuous output neurons.
The student as well as the teacher network have three hidden units
with weights $\boldsymbol{W}_i$ and $\boldsymbol{B}_i$, the transfer
function of the hidden units is the error function.  For simplicity
the output neuron is linear with fixed weights
\begin{equation}\label{siebenunddreissig}
\sigma = \sum\limits^3_{i=1} \;\; \mbox{erf} (\boldsymbol{W}_i \; \boldsymbol{S}/\sqrt{2})
\end{equation}
The teacher presents $\alpha N$ many examples given by
\begin{equation}\label{achtunddreissig}
\sigma^{\nu}_B = \sum\limits^3_{i=1} \;\; \mbox{erf} (\boldsymbol{B}_i \;
\boldsymbol{S}^{\nu}/\sqrt{2})
\end{equation}
The error of a single example is defined as the quadratic deviation
\begin{equation}\label{neununddreissig}
\varepsilon (\boldsymbol{W}_1, \boldsymbol{W}_2, \boldsymbol{W}_3,
\boldsymbol{S}^{\nu}) = \frac{1}{2} (\sigma^{\nu} - \sigma^{\nu}_{B} )^2
\end{equation}
In analogy to backpropagation [Hertz et al 1991], the change of
weights is proportional to the gradient of the training error of a
single example:
\begin{equation}\label{vierzig}
\boldsymbol{W}_k (\nu + 1) = \boldsymbol{W}_k (\nu) - \frac{\eta}{N} \;\;
\vec{\nabla}_{\boldsymbol{W}_{k}} \;\; \varepsilon
\end{equation}
From this equation a system of first order, nonlinear and coupled differential equations can be
derived for the set of order parameters
\begin{eqnarray}\label{einundvierzig}
R_{jk} & = & \frac{1}{N} \; \boldsymbol{W}_j \; \boldsymbol{B}_k \nonumber \\
Q_{jk} & = & \frac{1}{N} \; \boldsymbol{W}_j \; \boldsymbol{W}_k
\end{eqnarray}

In our case there are 15 order parameters which change after each presentation of a new
example. In the limit of $N \rightarrow \infty$ the index $\nu$ becomes a continuous ''time"
$\alpha$. Hence, one has to calculate the flow of $R_{jk} (\alpha), Q_{jk} (\alpha)$ in the 15
dimensional space of order parameters which determine the generalization error $\varepsilon_g
(\alpha)$.
\begin{figure}[h]
\centerline{
\epsfysize=4cm
\epsffile{fig14.ps}
}
\caption{ Generalization error as a function of the size of training set for a two layer
network (schematic). The plateau is related to a fixed point with one weak repulsive direction
for the flow of order parameters. (From Biehl et al 1997) }
\end{figure}

It turns out that there are several fixed points of this flow, which have important consequences
for the behavior of the generalization error. Fig.~14 shows a typical example. For a small
number of examples, $\varepsilon_g$ decreases fast. But then the generalization error almost
does not change for a long training period. Suddenly it decreases to good performance of the
network.

This plateau of $\varepsilon_g (\alpha)$ which is observed in applications, too, can be
understood in terms of the flow of order parameters [Biehl et al 1997]. There is one fixed point which is
strongly attractive in almost all directions. But in one or a few directions it has a small repulsive
component. Hence, the flow remains for quite a while (depending on initial conditions) close to
this fixed point with a large generalization error, but then it flows away to the completely
attractive fixed point with zero error.

The number of fixed points depends on the learning rate $\eta$ of the trainig rule. In our simple example there are at least ten different fixed points for $\eta=1$.
With increasing $\eta$ some fixed points  split into two, which
usually means that some symmetry is broken. Fixed points  suddenly appear, disappear or
annihilate with varying learning rate $\eta$.  Such discontinuous
behavior is reflected in the generalization error $\varepsilon_g
(\alpha)$.

\section{Noise estimation}

The examples, given by a teacher network, may have errors. To what extend can a student
network derive information about the teacher weights from a set of faulty examples? This
problem has been investigated in detail [Copelli et al 1997]. We consider a committee machine with a
tree architecture
\begin{equation}\label{zweiundvierzig}
\sigma = \mbox{sign} [ \sum\limits^{K}_{i=1} \;  \mbox{sign} \; \boldsymbol{W}_i \;
\boldsymbol{S}_i ]
\end{equation}
The student as well as the teacher network have the same number $K$ of hidden units. The
examples are distorted by noise: The bit $\sigma^{\nu}_B$ is flipped with probability
$\lambda$. For $\lambda = 1/2$ there is
no information in the examples, but for $0 < \lambda < 1/2$ the student network may obtain an
overlap to the teacher one with increasing number $\alpha$ of examples.

We study the training algorithm
\begin{equation}\label{dreiundvierzig}
\boldsymbol{W}_k (\nu + 1) = \boldsymbol{W}_k (\nu) + \frac{1}{N} F_k
\boldsymbol{S}_k
\end{equation}
Instead of a parameter $\eta$ we have introduced a function $F_k$
which is determined from a variational principle which maximizes the
decrease of generalization error $\varepsilon_g$ at each training step
[Kinouchi and Caticha 1992]. Hence, one can define an optimal weight
change, which depends on the order parameters. It also contains the noise rate $\lambda$ which is
not known in general; it has to be estimated by a value $\Lambda$,
$F_k (\lambda)$ is replaced by $F_k (\Lambda)$.
\begin{figure}[h]
\centerline{
\begin{turn}{-90}
\epsfxsize=8cm
\epsffile{fig15.ps}
\end{turn}
}
\caption{ On--line training with optimal weights. $\Lambda$ is the estimated and
$\lambda$ the true noise level of the training examples. Above the solid line perfect
generalization is possible, $\varepsilon_g (\alpha \rightarrow \infty) = 0$. Below the dashed
line the network cannot generalize at all. Between the dashed and solid line a partial
recognition of the teacher network is possible. (From Copelli et al, 1997) }
\end{figure}

The generalization error has been calculated for optimal on--line learning [Copelli et al 1997],
Fig.~15 shows the result. There are sharp boundaries in the $(\lambda, \Lambda)$ plane where
the behavior of the network changes drastically. Fastest decay of $\varepsilon_g (\alpha)$ is
obtained if the true noise is estimated correctly, $\lambda = \Lambda$, as expected. If the
estimated noise parameter $\lambda$ is large enough, then $\varepsilon_g (\alpha)$ still
decreases to zero in the limit of an infinite number $\alpha$ of examples. However, if
$\Lambda$
is below the dashed line, then the network cannot generalize at all. If $\Lambda$ lies in
the
intermediate region then the generalization error decreases to a nonzero value; the network can
generalize only partially. Again we observe sudden changes in the behavior of the network as
a function of model parameters.

\section{Time series generalization}

Most of the work on the statistical physics of neural networks has been done on static data. A
set of input vectors $\{\boldsymbol{S}^{\nu} \}$ is taken from a static distribution and
classification labels $\{ \sigma^{\nu}\}$ are taken from a static rule. Only recently this research
program has been extended to the analysis of time series [Eisenstein et al, 1995], which is an
important field of
applications of neural networks.

\begin{figure}[h]
\centerline{
\epsfysize=2cm
\epsffile{fig16.ps}
}
\caption{ A perceptron working as sequence generator. The sequence $S_l$ of numbers
is generated by a perceptron moving to the right. }
\end{figure}

In the simplest case our elementary unit, the perceptron, is trained to a sequence $S_i$ of real
numbers, where $i$ is a discrete time. As shown in Fig.~16, the perceptron takes a window of
$N$ numbers as input and makes a prediction of the following number,
\begin{equation}\label{vierundvierzig}
S_l^{\prime} = \tanh [ \frac{\beta}{N} \; \sum\limits^N_{j = 1} \; W_j \; S_{l - j} ]
\end{equation}
In the training phase the weights $\boldsymbol{W}$ are changed to decrease the error
$(S_l^{\prime} - S_l)^2 $.

In order to apply the concepts of statistical physics one needs a well defined sequence
$\{S_i\}$. As before it may be given by a teacher perceptron with weight vector
$\boldsymbol{B}$. For a given window of $N$ numbers $(S_{l-N, ... , } S_{l-1})$ the
perceptron defines the number $S_l$. Then it moves one step and generates $S_{l+1}$. It
turns out that the generation of time series is already an interesting problem with many
unsolved puzzles [Eisenstein et al 1995, Kanter et al 1995, Schr\"oder and Kinzel 1997].

The numerical investigation of the sequence generator shows that an initial state of
Eq~(\ref{vierundvierzig}) approaches a quasi periodic attractor which is related to a peak in the
Fourier
spectrum of $\boldsymbol{W}$ [Eisenstein et al 1995]. Hence, the perceptron selects one
mode of the
couplings. Therefore it is useful to study couplings with a single Fourier component
\begin{equation}\label{fuenfundvierzig}
W_j = \cos (2 \pi k \; \frac{j}{N} - \pi \phi )
\end{equation}
$k$ is the frequency and $\phi$ the phase of the weights. An attractor of
Eq.~(\ref{vierundvierzig}) is the solution of
\begin{equation}\label{sechsundvierzig}
S_l = \tanh [ \frac{\beta}{N} \; \sum\limits^N_{j=1} \, \cos (2\pi k \frac{j}{N} - \pi \phi) S_{l-
j}]
\end{equation}
Recently this equation could be solved analytically [Kanter et al 1995]: For small values of
$\beta$ the
attractor is zero, $S_l = 0$. For a critical value, which is independent of the frequency $k$,
\begin{equation}\label{siebenundvierzig}
\beta_c = 2 \frac{\pi \phi}{\sin \pi \phi}
\end{equation}
there appears the solution
\begin{equation}\label{achtundvierzig}
S_l = \tanh [A(\beta) \; \cos(2\pi (k + \phi) \frac{l}{N} )]
\end{equation}
The phase $\phi$ of the weights shifts the frequency $k + \phi$ of the solution. The amplitude
$A(\beta)$ is a continuous function of the distance $\beta - \beta_c > 0$ to the critical point.
For $\beta \rightarrow \infty \;\; \; \tanh$ is replaced by sign and the sequence generator
becomes a bit generator. In this case the solution, Eq.~(\ref{achtundvierzig}), is more
complicated, but again the phase $\phi$ shifts the frequency of the bit sequence [Schr\"oder
and Kinzel 1997].
\begin{figure}[h]
\centerline{
\epsfxsize=8cm
\epsffile{fig17.eps}
}
\caption{ Return map of a sequence generated by a multilayer network with two hidden
units. The one--dimensional attractor in the middle becomes a two--dimensional one, if the
parameter $\beta$ is increased. (From Kanter et al, 1995) }
\end{figure}

If the network is a multilayer perceptron with $K$ hidden units, then
the attractor of the sequence generator is a kind of superposition of
the single modes of each hidden unit [Kanter et al 1995].  Each unit
has ist own critical point and the number of nonzero solution
determines the dimension of the attractor. This is shown in the return
map of Fig.~17 where $S_{l+1}$ is plotted against $S_l$
for $K=2$ hidden units. Increasing $\beta$ first one component is
activated leading to a one dimensional attractor shown in the middle
of the figure. Since in general the frequency is irrational, the
attractor is a continuous curve. For larger values of $\beta$ the
second unit is activated, giving the two--dimensional attractor with
larger amplitude.

\section{Summary}

Concepts of statistical physics have successfully been applied to the theory of neural
computation. The cooperative behavior of a large number of interacting neurons can be
described in terms of partition sums and order parameters. The competition between training
error and entropy may lead to discontinuous properties.\\
The approach of statistical mechanics has several advantages:
\begin{enumerate}
\item  Networks with an infinite number of neurons and synapses can be calculated analytically.
Complex cooperative behavior of interacting neurons is described in terms of a few order
parameters.
\item The results are obtained for a typical situation, for instance for the most general set of
examples.
\item One obtains exact mathematical relations between the observed cooperative properties of
the network, its model parameters and the size of the training set.
\item Many networks and algorithms show discontinuous properties as a function of model
parameters or the number of presented examples. Statistical physics can describe such sudden
changes in the cooperative behavior of the network.
\end{enumerate}
Statistical mechanics of neural networks has been applied to a variety of problems; we just
want to mention learning from examples, generalization, associative memory, attractor
networks, structure recognition, clustering, classification, coding and time series analysis. For
all of these problems general properties have been calculated mathematically. Novel and
unexpected results have been found. Hence, I think that in the last 15 years theoretical physics
has successfully contributed to our understanding of neural networks, with impact on
neurobiology and computer science.

{\bf Acknowledgement:} The author thanks Michael Biehl for comments on the manuscript.

{\large \bf Literature}

Hertz, J.~A., A.~Krogh and R.~G.~Palmer, 1991, Introduction to the Theory of Neural
Computation (Addison Wesley)

Amit, D.~, H.~Gutfreund and H.~Sompolinsky, 1987, Annals of Physics, {\bf 173}, 30-67

Barkai, E.~, D.~Hansel and I.~Kanter, 1990, Phys.~Rev.~Lett.~{\bf 65} 2312

Barkai, N.~and H.~Sompolinsky, 1994, Phys.~Rev.~{\bf E 50}, 1766

Biehl, M.~and A.~Mietzner 1993, Europhys.~Lett.~{\bf 24}, 421--426

Biehl, M.~and H.~Schwarze, 1995, J.~Phys.~ {\bf A 28}, 643(?)

Biehl, M.~, 1997, lecture notes, University of W\"urzburg

Biehl, M.~, P.~Riegler and C.~W\"ohler, 1997, to be published

Copelli, M.~, R.~Eichhorn, O.~Kinouchi, M.~Biehl, R.~Simonetti, P.~Riegler and N.~Caticha,
1997, Europhys.~Lett.~{\bf 37}, 427-432

Eisenstein, E.~, I.~Kanter, D.~Kessler and W.~Kinzel, 1995, Phys.~Rev.~Lett.~{\bf  74}, 6

Fischer, K.~H.~and J.~A.~Hertz, 1991, Spin Glasses, (Cambridge University Press)

Gardner, E.~, 1988, J.~Phys.~{\bf A 21}, 257--270

Hopfield, J.~J.~, 1982, Proceedings of the National Academy of Sciences, USA {\bf 79},
2554--2558

Kanter, I.~, D.~A.~Kessler, A.~Priel and E.~Eisenstein,  1995, Phys.~Rev.~Lett.~{\bf 75},
2614--2617

Kanter, I. and H. Sompolinsky, 1987, Phys. Rev. {\bf A 35}, 380

Kinzel, W. , 1985, Z. Physik {\bf B60}, 205

Kinouchi, O.~and N.~Caticha, 1992, J.~Phys.~{\bf A 25}, 6243

Oja, E.~, 1982, J.~Math.~Biol.~{\bf 15}, 267

Opper, M.~, 1994, Phys.~Rev.~Lett.~{\bf 72}, 2113

Opper, M.~ and W.~Kinzel, 1996, in Models of Neural Networks III, ed.~by E.~Domany,
J.~L.~van Hemmen and K.~Schulten (Springer, Berlin)

Saad, D.~ and S.~A.~Solla, 1995, Phys.~Rev.~Lett. {\bf 74}, 4337

Schl\"afli, L.~, 1852, Theorie der vielfachen Kontinuit\"at, Gesammelte Mathematische
Abhandlungen, ed.~Steiner--Schl\"afli--Komittee Basel, Birkh\"auser p 171

Schr\"oder, M.~and W.~Kinzel, 1997, to be published

Schwarze, H.~, M.~Opper and W.~Kinzel, 1992, Phys.~Rev.~{\bf A 46}, R 6185

Seung, H.~S.~, H.~Sompolinsky and N.~Tishby, 1992, Phys. Rev. {\bf A 45}, 6056

Watkin, T.~L.~H.~, A.~Rau and M. Biehl, 1993, Rev.~ Mod.~Phys.~ {\bf 65}, 499

Watkin, T.~L.~H.~and J.~--P.~Nadal, 1994, J.~Phys.~{\bf A 27}, 1889

Yeomans, J.~, 1992, Statistical mechanics of phase transitions, (Clarenden Press, Oxford)

\end{document}

redpajama_set_name: RedPajamaArXiv


Creamy Mushroom curry / Malai Methi Mushroom. Delicate creamy curry, with the creaminess of cashew butter and aromatic fenugreek leaves, mild spiced, served with fragrant saffron basmati pilaf.
As I grow older I want to live a more gratifying life. We consume not just food but a lot more than we realize. I have slowly and intentionally tried to weed out the 'unwanted' in the pursuit of a meaningful existence. There is so much to weed out, externally and internally, it is amazing the amount of junk we accumulate. Weeding out the external was hard but nothing compared to weeding out and being mindful of my internal feelings and doings. I do not let anyone's thoughts or perception, color my own thought, I take it as a commentary of the moment and let it go.
The internal 'noise' is something else, it is your own junk to clean and keep up. The best thing I did a few months back was to upload 2 apps on my phone, one headspace and the other, moment. I am not going to tell you what it is, but both of them are great tools that helped me reduce the noise inside my head. No, this is not a paid endorsements for these apps. For me walking helps me focus like nothing else does. This year when I went to write down my goals, not surprisingly, all I had was just one word : gratification. That encompassed so much, I'm not competing with anyone but myself. Neither am I proving my metal to anyone but myself, for in the end I have to be answerable to no one but myself.
As I was wondering around the forest preserve, I saw this patch of daffodils...Daffodils have almost become synonymous with Wordsworth.
Sharing all this poetry, I was almost lost ... and forgot I have to write a recipe. Scroll down and you will find it. Mushroom was one of my dad's favorite. Teen is very fond of it as well, and needless to say, it is definitely my top ten favorite. The Mr. , has acquired taste for it and am happy to say loves it as well. This is served with saffron rice pilaf, I am going to do a separate post for the pilaf, reason being, if I pair it with any-other curry in future posts, it is easy to link the recipe.
Mix the vinegar and water, soak the mushroom in vinegar-water. For about 20 minutes, it marinates the mushrooms a bit but more than that, it cleans it up really well.
Soak the cashew nut and poppy seed in water - NO vinegar.
Clean the mushroom and chop it in quarters. Heat a pan, add butter (1 tbsp. ) and sauté the mushrooms, add salt and keep aside.
In another pan, melt the remaining butter add cumin seeds and let it splatter, add onions, turmeric root (grated) or garlic and roast till mild golden color, let it not be brown as it will ruin the color of the curry.
Pour in the tomato puree, along with salt and fenugreek leaves. Roast till the fenugreek is cooked. You can substitute spinach here, but I love the flavor of fenugreek leaves, if you have not tried it... you must!
Make a fine paste from the cashew you have soaked. Using poppy seeds is optional - it has a very mild flavor but more importantly it thickens the gravy.
Add the red chili powder, garam masala and sauté. Now pour the cashew paste and cream and stir.
Drain the mushroom, it will leave water and add the mushroom to the gravy.
Let it simmer for at least 20-30 minutes on low flame, stir occasionally.
Serve with Saffron rice - recipe coming soon.


text:
Dasychira ruandana is een donsvlinder uit de familie van de spinneruilen (Erebidae). De wetenschappelijke naam van de soort is voor het eerst geldig gepubliceerd door Grünberg.

ruandana

redpajama_set_name: RedPajamaWikipedia


Nike's latest version of the legendary Air Max 97 is travelling at its own leisurely pace, one of earth's fastest silhouettes adorned in flannelette for Fall.
Originally modelled on the Japanese Shinkansen, the Air Max 97 has seen a blinding number of iterations. With Nike's latest build, the the Swoosh have opted for a 'Sail' and black build, giving the silhouette a flatter, more unified aesthetic.
The 'Sail'/black Nike Air Max 97 is available now for $170.


text:
Засте́нки () — деревня в Зубовском сельсовете Оршанского района Витебской области Белоруссии.

География
Деревня Застенки находится в 19 км на юг от Орши, и в 13 км на северо-восток от Шклова. Расположена на автодороге М-8 Е-95.

История
В 1901 году в деревне Застенки был найден клад из 1200 арабских монет VIII в., что может свидетельствовать о прохождении некогда через Застенки торгового волока (из Орши в Витебск, который из Днепра через Оршу вёл в Западную Двину, т. н. водный путь «из варяг в греки»).

1 сентября 1907 года в деревне сильный пожар уничтожил 28 домов.

В 1910 году деревня относилась к Оршанскому уезду Могилёвской губернии, волость Баранская

После революции деревня относилась к Копыскому району.

Достопримечательности

На территории деревни сохранилась почтовая станция середины XIX века

Отдых
В деревне для отдыха туристов находится сельская усадьба площадью 0,65 га.

Имеется кафе придорожного сервиса «4 сезона», с обустроенной стоянкой (20 мест) и мотелем (10 мест).

Знаменитые уроженцы и жители
 Ловенецкий Пётр Людвигович (1914 г. р. (1911 ?), д. Застенки, Копысский р-н, Белорусия; белорус). Проживал в г. Нижний Тагил, работал в «Тагилстрое» паровщиком. 6 августа 1937 года был арестован, 1 октября 1937 года осужден тройкой УНКВД Свердловской области по ст. 58-10 за участие в антисоветских террористических организациях. Приговор: ВМН. Расстрелян. Реабилитирован посмертно 11 мая 1989 года Указом ПВС СССР. Номер дела: 41215-П

Примечания

Литература

Населённые пункты Оршанского района (Витебская область)


The Advanced technique abortion procedure is a preferred technique to give sedation for a stress free safe abortion procedure at the Family hospital at Mumbai India.
An advanced technique of sedation which is safe and painless. It causes the woman to gradually go to sleep and allow the procedure to be done in a safe and pleasant manner. No bad experience and no after effects.
The women is needed to be fasting for about 4 hours. After the pre medications, the woman is taken to the Operation room. The monitors are placed on the chest, the oxygen nasal canula is placed in the nose, a small injection is given in the fore arm and the woman is dreaming!
It is as simple as that. No pain. No unpleasant experience.
She wakes up in the room gradually and we assure you sweet dreams.
So the term - Sweet Dream Anasthesia.


text:
Audio lessons
Pronunciation audio
Prices and Salaries
Russian police ranks
Russian Police Ranks and Insignia
Russian police ranks are divided into four groups:
The first group is senior NCOs: sergeants and warrant officers.
The second group is junior officers: lieutenants and captains.
The third group is senior officers: majors, lieutenant colonels, and colonels.
The fourth group is general officers: major generals, lieutenant generals, colonel generals, and generals of police.
The police ranks are conferred for life. When a police officer retires, the words «in retirement» are added to their rank (for example, a captain in retirement, a major in retirement, a colonel in retirement, and so on).
Russian police ranks are conferred sequentially. A police officer should serve a certain fixed term. In Russia, all ranks are divided into first and regular.
Getting a regular title is not so easy. When it is conferred, special attention is paid to the officer's education. But sometimes, a police officer can be promoted ahead of time, for example, for special service to the country. However, if the officer has served in the previous rank for less than half of the required period, this promotion is postponed.
All Russian police ranks, from cadets to generals of police have their insignia. But how can one determine a police officer's rank and position? Here is a table:
The table below has become a little obsolete and undergone some insignificant changes: there is no «police» emblem near the button, and the first sergeant's strap has a wide stripe, which is longer than that on the new strap (one can see it on the above photo). The shoulder strap of the general of police has changed its color. Other things are the same.
The shoulder straps have distinctive signs: a button and a «police» emblem next to it. Police cadets also have a letter K.
Senior NCOs: Sergeants and Warrant Officers
Junior sergeants, sergeants, and senior sergeants have stripes across their shoulder straps. A junior sergeant has two stripes, a sergeant has three stripes, a senior sergeant has one wide stripe across their strap, while a first sergeant has the same wide stripe but along their strap.
There are small vertically‐arranged stars without any stripes on warrant officers' shoulder straps. Warrant officers have two stars, while chief warrant officers have three stars.
Junior Officers: Lieutenants and Captains
The straps have a vertical red stripe. The stars are small. A junior lieutenant has one star on a red stripe, a lieutenant has two stars with a stripe between them, a senior lieutenant has three stars, two across the stripe and one on the stripe, a captain has four stars, two across the stripe and two on the stripe.
Senior Officers: Majors, Lieutenant Colonels, and Colonels
There are two red vertical stripes and big stars along their shoulder straps. One star in the middle inside the stripes indicates a major. Two stars located on the stripes means a lieutenant colonel. Three stars, two on the stripes and one between them, are a sign of a colonel.
General Officers: Generals and Generals of Police
The stars are big without any stripes. They are arranged vertically. A major general has one star, a lieutenant general has two stars, and a colonel general has three stars. One bigger star with a large Russian coat of arms belongs to a general of police.
Russian police ranks are conferred after a certain period of service.
Privates and junior sergeants need to serve for 1 year, sergeants – for 2 years, and senior sergeants – for 3 years. Warrant officers should serve for 5 years to become junior lieutenant. A junior lieutenant's period of service is 1 year, a lieutenant's – 2 years, a senior lieutenant's and a captain's – 3 years, a major's – 4 years. Lieutenant colonels should serve for 5 years.
Colonels and generals are promoted only for special merit and service to their country. This rule does not apply to the highest rank, general of police.
Academy Stars
HSK Standard Course
Sejong Korean Textbook
Popular lessons
German audio lessons
German video lessons
French audio lessons
French video lessons
Italian audio lessons
Italian video lessons
Spanish audio lessons
Spanish video lessons
Portuguese audio lessons
Portuguese video lessons
Russian sayings
Russian riddles
Russian words
Russian idioms
Russian verbs
Russian flashcards
Russian audio lessons
Russian video lessons
History Russian language
Prices in Russia
Prices in Belarus
Prices in Ukraine
Prices in Bulgaria
Interesting languages
Klingon language
Elvish language
Gypsy language
Maori language
Na'vi language
© 2022 - Foreign languages

redpajama_set_name: RedPajamaCommonCrawl


People who believe their success is based on hard work, learning, training, and perseverance are said to have a "growth mindset." I want to describe 4 specific examples of how you can apply this mindset in your job today.
When hiring new staff (by the way, I might be looking for you), I want to meet people that are just nuts about growth. They should have great personalities and be funny, with a touch of craziness. Furthermore, they should be team-players with strong conceptual, analytical and social skills, able to work independently and have an entrepreneurial ability. And last but not least, one of the obligatory skills is having a growth mindset.
Do not feel offended by criticism from a manager. Instead, use this feedback to get better at your job.
You may have to accomplish a new task, and be a little insecure. In this instance, do not think: "I will never succeed", but be confident that you will learn the new skills that you will need.
If you are a manager yourself, opt for task rotation or even job rotation. That way your employees will continue to develop their skill sets.
When hiring new staff, pay attention to the ability of people to learn new things.
I will be honest with you: the reason that I value a growth mindset in my people, is because it has helped me to accomplish difficult tasks many times. Every time I thought that I was not ready for an assignment, this way of thinking saved me.
So, I recommend everyone to foster a growth mindset. At RevelX, we can help you accomplish this, for example, in one of our Growth Mindset Workshops.


text:
INSTITUTE RECYCLES INSEPARABLE AUTO FLUFF
Paul Gargaro
University of Detroit Mercy researchers want to kick the fluff out of the car-disposal industry. Recyclers say it's about time.
For years, questions of utility surrounded fluff, the industry moniker for the nonmetal residue that remains after cars are shredded. It comprises plastics, rubber and upholstery and is widely regarded as useless because it is difficult to separate for recycling.
After 15 months of research, however, scientists at UDM's Polymer Institute say they have found a potentially profitable life for fluff and its cousin, automo-tive shredder residue, or ASR.
``The focus has always been on recycling like materials,'' said Kurt Frisch, director of research at the Polymer Institute. ``We're now taking whatever is left and recycling it for a number of very marketable uses.''
Potential uses range from railroad ties to lightweight cushioning. The UDM labs are creating a range of products, some with the pliability of foam rubber, others with compression strengths three times that of concrete. The materials used to bind the fluff can cost as little as 30 cents a pound, which UDM officials say makes this an economical recycling option.
The key to making these products is understanding fluff and ASR at their origins in auto shredders.
Fluff is one of three byproducts created when a car is shredded. The others are ferrous metals, which contain iron; and nonferrous shredder residue, which consists of nonferrous metals, plastics and rubber.
Nonferrous residue is further broken down into ASR. The terms fluff and ASR often are used interchangeably.
The amount of nonferrous materials in cars has risen in recent years as manufacturers have strived to build lighter and more efficient models.
Roughly 25 percent of today's cars, except for tires, end up in landfills.
Sandy Labana, chairman of the Big Three's Vehicle Recycling Partnership, said waste accounts for about 3 million tons annually. Although that sounds like a lot, Labana said, it represents about 2 percent of the waste municipalities send to landfills.
Nevertheless, said Labana, automakers have an economic interest in salvaging as much material as possible.
``The driving force is that these materials have pretty good value, so they should be recycled and reused,'' he said. ``That's the most logical reason.''
Test materials for UDM's research were supplied by Belle-ville's Huron Valley Steel Co., a leading reclaimer and processor of nonferrous automotive metals.
Richard Osterberg, Huron Valley's vice president of operations, said the debate over what to do with automotive residue has raged for years. Researchers have analyzed the merits of converting it into fuel as either oil or burnable gas, he said. Others have extracted materials from the scrap and reconfigured them into polymers that can make new products.
The results of the UDM tests are promising, according to Osterberg.
``The thing that excited me most about UDM's approach was that they weren't attempting to sort out the fluff,'' he said. ``Their approach was to take it the way it is and create some sort of composite polymer using different kinds of binder material.''
The UDM team is able to do this because it has identified a variety of shredding processes. Some shredders use wet processes; others dry, forced-air systems. Few are exactly the same.
By understanding how and at which point material is broken apart, UDM scientists say, residue particles can be classified by size and consistency. That makes future use much easier.
``The big misconception is that there's one waste stream - one ASR,'' said Vahid Sendijarevic, an associate research professor at the Polymer Institute and the principal investigator for the project. ``But ASR varies depending on how it's separated at the shredder. We can now identify which shredder process the ASR came from ... and make different materials depending on its origin.''
UDM's work is so promising it was named the year's best recycling research project by the Michigan Materials and Processing Institute, a nonprofit consortium of industries and universities that work on polymer composite materials for the durable-goods industry. The institute is also a contributor to UDM's research grant.
Osterberg of Huron Valley Steel said he agrees with the designation.
``I think they have demonstrated the most practical approach,'' he said. ``A lot of times, universities talk pie-in-the-sky stuff, but this really appears to be a practical solution.''
UDM next month is slated to share its findings with industry, including representatives from the Big Three.


text:
Filip Richtermoc (* 9. června 1971 Hořice v Podkrkonoší) je český divadelní a filmový herec, člen uměleckého souboru Klicperova divadla v Hradci Králové od sezony 1995/96. Vystudoval JAMU v Brně. Svoji kariéru začal po maturitě jako elév v Horáckém divadle v Jihlavě a již během své první divadelní sezóny byl přijat na JAMU. Během svých studií hostoval v Národním divadle Brno v inscenaci Timon Aténský. V Klicperově divadle v Hradci Králové zazářil například v představení Petrolejové lampy a zahrál si v celé řadě dalších rolí: Tybalta v Romeovi a Julii, Jacoba v Třígrošové opěře, Dona Juana, Tuzenbacha ve Třech sestrách, Kajetána v Akvabelách, Chlestakova v Revizorovi, Cassiela Wildemanna v Mefistovi, Alberta Gregora ve Věci Makropulos, Sorina v Rackovi, Jardu Kýbla v Hoří, má panenko!, Petrtýla ve Světácích či Alžbětu, manželku krále Edwarda IV., v inscenaci Richard III. V Brně účinkoval v představeních HaDivadla a Divadla bratří Mrštíků. Vedle rolí v televizních seriálech vystupuje i ve filmech (Po letech, 1995; Narušitel, 2019).

Odkazy

Reference

Čeští divadelní herci
Čeští filmoví herci
Narození 9. června
Narození v roce 1971
Narození v Hořicích
Žijící lidé
Muži
Absolventi Divadelní fakulty Janáčkovy akademie múzických umění


text:
CIM is Changing the Face of Classical Music: 235 percent increase in African American and Latinx students since Fall 2016
Preparatory & Continuing Education
From left, student trombonists Christopher Wengert-Ramos, Philip Williams, Mary Cubero Navarro and Christopher Jordan
Following national College Signing Day on May 1, the Cleveland Institute of Music has seated 118 of the world's most talented classical music students for fall 2019, with 13 percent of the entering students coming from historically underrepresented racial groups. The 15 incoming African American and Latinx students join their peers at CIM where the diversity of the student body has increased by a remarkable 235 percent to an historic high: 15 percent of the 375 students.
"CIM has never shied away from addressing the inequities in classical music and in the conservatories and schools of music who prepare the next generation of musicians," said Paul W. Hogle, CIM's president and CEO. "In the highly competitive world of America's top classical music conservatories, students and their families put their trust in programs because of the faculty, the learning environment and the school's collaborators. This is transformational work that will influence generations of students."
This work includes:
Establishing CIM's Musical Pathway Fellowship (MPF), which provides concentrated pre-college opportunities for aspiring young Cleveland-area African American and Latinx classical musicians.
A student-formed Black Student Union, whose purpose is to foster and encourage meaningful dialogue and fellowship among students, faculty and staff.
Partnering with the Sphinx Organization, a Detroit-based nonprofit organization dedicated to the development of young African American and Latinx classical musicians and leaders.
Working with the Atlanta Symphony Orchestra's Talent Development Program (TDP) – whose generation-long commitment to training African American and Latinx classical music students for consistent acceptance in top music programs is recognized as a national model of excellence
"The way to change the face of classical music is to recruit, empower and engage a community of exceptionally well-prepared student musicians of color and immerse them in CIM's high-quality training and the subsequent benefits that training provides," Hogle said. "CIM has become a leading voice in the national conversation among conservatories and music schools, and takes accountability for increasing diversity in our student populations. We are the gatekeepers of the industry, and, if we are to help address the lack of diversity on our stages, it's up to us to start with our classrooms and our practice rooms. Our aim is to change the face of conservatories, music schools and classical music so we look more like the communities we serve as we pursue the future of classical music."
Jerrod Price, CIM's associate dean of admissions and enrollment management and executive director of preparatory, enrollment and pathway programs, attributed the Institute's enrollment success to CIM's reputation for excellence, commitment to lowering the net cost of attendance and leadership in creating opportunities for exceptionally talented young musicians of color.
"Our overarching goal is to enable all of our students to pursue their dreams," said Price. "Through a commitment to offering better access to a classical music education and making students even better, lifelong musicians, we are ensuring that CIM is a place where they can grow and succeed."
Philip Williams, a CIM freshman trombonist from Atlanta and alumnus of TDP, says he chose to attend CIM because of the school's inviting atmosphere. Williams, 19, is one of the founders of the new Black Student Union. He and three fellow students started the organization to get involved and help create more of the connections all students seek at college.
He emphasizes, however, that while the CIM Black Student Union is a place for the Institute's African American students to gather and discuss important issues, any CIM student is invited to join.
"We have created a place on campus where students can convene in a safe space to talk about anything they want," Williams said. "It's important to keep in mind that all students are welcome. CIM has made great strides in the change it seeks. Once we get where we want to be, we should not let up."
Hogle agrees. "CIM will continue to engage its students, faculty, staff and trustees in a lifestyle of diversity, equity and inclusion, and implement practices and programs to meet the CIM community's diverse and ever-changing needs," he said.
The incoming class of 2023 is one of exceptional artistic distinction, academic achievement and geographic reach. They are competition prizewinners, alumni of premier music festivals and some of the country's finest youth orchestras, and several possess an entrepreneurial spirit as founders or co-founders of their own organizations and businesses. Musical accomplishments of the class include:
Prizewinners from such prestigious events as the Vendome Prize, American Protégé International Vocal Competition, Calgary Music Festival, Young Concert Artists International Auditions, Italian Scriabin International Piano Competition and the Hong Kong International Music Festival Competition.
Professional members of the New York String Orchestra, Erie Philharmonic, Seattle Symphony, Canton Symphony and Tallahassee Symphony Orchestra, among others; and major youth orchestras in Atlanta, Cleveland, Chicago, Cincinnati, New York City, Philadelphia, San Francisco and St. Louis, as well as National Youth Orchestra 2.
Alumni of elite music festivals, including Aspen Music Festival and School, Heifetz International Music Institute, Sphinx Performance Academy, Boston University Tanglewood Institute, National Repertory Orchestra, Bowdoin International Music Festival and the Beijing International Music Festival and Academy.
Other compelling statistics include:
International students from 13 different countries comprise 30 percent of the class; domestically, students hail from 27 states
18 percent of the class are first-generation college students
Seven of the 63 undergraduates are from the Interlochen Arts Academy
Five students are enrolling in a dual degree program with Case Western Reserve University
13 students from the incoming class are returning to CIM for additional study, two of whom are graduates of CIM's Young Artist Program
"I am eager to welcome this impressive group to their new home in Cleveland, in the heart of University Circle," Hogle said. "Seeing – and hearing – the ways in which they will enrich our lives is going to be a privilege and an honor. And I am delighted that they have accepted our invitation to fulfill their dreams and potential at CIM, and to become part of this very special place."


never failing to touch the soul behind my eyes.
finding nothing I am searching for.
and I feel you in the breeze.
Tears become my odes to you.
but makes me smile through a child's giggles.
knowing that my feet will always pick up your beat.
Under the stars, I am yours.
the thunder that rumbles on a summer's night.
calming the fire and urging me to burn brighter.
you always know the secrets i won't tell.
With you, I cannot hide.
With you, I will not run.


Democrats in Congress are pushing legislation that would bring back the Federal Communications Commission's (FCC) net neutrality rules.
Reps. Henry Waxman (D-Calif.) and Anna Eshoo (D-Calif.) and Sen. Ed Markey Edward (Ed) John MarkeyBen & Jerry's backs Green New Deal: 'We have to act now' Warren praises Ocasio-Cortez in Time 100 The Hill's 12:30 Report: Trump, Dems prep for Mueller report's release MORE (D-Mass.) led a dozen other Democrats in calling for the rules to be reinstated to prevent Internet service providers from treating websites differently online.
"The Internet is an engine of economic growth because it has always been an open platform for competition and innovation," said Waxman, the top Democrat on the Energy and Commerce Committee, in a statement.
"Our bill very simply ensures that consumers can continue to access the content and applications of their choosing online.
The Open Internet Preservation Act would restore the FCC regulations struck down by a federal appeals court earlier this month until the commission decided to take new actions on the issue.
The bill is likely to meet strong opposition from Republicans in Congress, who have said that the rules allowed the federal government to have too much influence in the operation of the Internet.
The FCC's net neutrality rules, called the Open Internet order, prevented Internet providers from blocking or slowing access to specific websites. Web providers have complained that popular websites like Netflix use large amounts of bandwidth and should be treated accordingly.
Consumer advocacy groups have said that the rules were critical to ensuring all content is treated fairly online.
Last month, the U.S. Court of Appeals for the D.C. Circuit overturned the FCC's rules. The decision was a major blow to the agency and President Obama, who had championed the rules.
"The FCC I know and Tom Wheeler are looking at all the options at their disposal. Potential appeals, potential rulemaking, a variety of tools that they may have in order to continue to vindicate the notion of a free and open Internet," he added.
The House Energy and Commerce Committee is working on updating the 1996 law regulating communications services.
Rep. Greg Walden (R-Ore.), head of the panel's Communications and Technology subcommittee, said in a statement that the new bill "will certainly be a part of the discussion" in that process.
"However, we remain vigorously opposed to any attempt to install the FCC as the traffic cop of the Internet," he said.


When you connect to our website, our web server may record the numerical Internet Protocol (IP) address of the computer you are using, information about your browser and operating system, date and time of access and the internet address and page which linked you to our site. The School's staff uses such information to help diagnose problems with our server, to administer and improve our website and to generate aggregate statistical reports that analyze general user information and demographics. Unless required by legal process, we do not link IP addresses to anything personally identifiable. This means that user sessions may be tracked by IP address, but a user's identity remains anonymous.


text:
Free hippie dating site
March 8, 2018| | 0 Comment
Guide to: Double Dating & The Last Day
Double Dating / Last Day of School
Double Dating & the Last Day
Watch Ned's Declassified Season 2 Episode 20 – Double Dating/Last Day of School
Double dating and the last day
Ned's declassified speed dating
DOUBLE DATING FOR FREE
Let's work together to keep the conversation civil. Get a grip on chromosomes. Perils of double dating Get a grip on chromosomes Teach and learn the love language Cuddling is all you do at this party. Select a City Close. Mumbai Mumbai search close. All Bombay Times print stories are available on. We serve personalized stories based on the selected city OK. The Times of India. Hazel Keech reveals her battle with depression and bulimia in 10yearchallenge.
Backpack Boy Kendre Berry is a student who carries an oversized guide including many various things. He has a crush on Claire Sawyer. Vanessa Johnson Spencer Locke is a mean, popular girl, seen only in Season 1, who cheats to get high grades and full awards at survival, Moze being her full victim, as revealed on the episode "Cheaters and Bullies". She also tries to humiliate Moze by stealing her diary, and in the episode "Notes" she gets Ned in trouble for passing notes.
Her character was replaced by Missy.
You may also stumble upon dating that looks different from your standard day/month/year or month/day/year styles. This can be confusing when you don't know.
Middle-school teens deal with daily dilemmas such as bullies, bad haircuts and weird teachers. The final dance of the year finds Ned and Suzie double-dating with Moze and an attractive boy named Jock Bitterman, while Cookie tries to manage escorting two different girlsneither of whom knows about the other. Later, it's the last day of school, and a robot created by Cookie wreaks havoc in the hallways. Christian Serratos Suzie Crabgrass. Sign up and add shows to get the latest updates about your favorite shows – Start Now.
Want to find a partner for sex? It is easy. Click here, registration is absolutely free!
Keep track of your favorite shows and movies, across all your devices. Sign up to get started Login About My Watchlist. Recommendations Discover Listings News. Watchlist Added Where to Watch. Back To: Episode Guide. First Aired: June 3, The final dance of the year finds Ned and Suzie double-dating with Moze and an attractive boy named Jock Bitterman, while Cookie tries to manage escorting two different girlsneither of whom knows about the other. Where to Watch. My News Sign up and add shows to get the latest updates about your favorite shows – Start Now.
Watch the video. Ned was always running to the bath room, and Moze was trying to help him and just when he went o kiss Suzie he kissed Moze instead. I watched that part a lot. Ned was afraid that Suzzie didn't want to date him and Moze would't want to be friends with him.
Double Dating; Last Day of School Part 1. Jun 3, Season-only. Season 2. Episode Ned goes on a double date to the final dance of the year. Season 3.
This is a truly latino dating the last day 7th grade girl. Guide guide season 2 at tv, reviews, and hunk jock bitterman. Deckinger's latest estimate the final dance, plot summary, ned does the short ned. Our darcy rose byrnes dating desktop experience. Comnickelodeon's ned's declassified and hunk jock bitterman. Later, and suzie double date. Both of neighborhood neds declassified double dating experience. Read about dating and hunk jock sees ned getting the same until his stomach acts up.
Ned's declassified school survival guide to all your memory. Stress: for the final dance and suzie double date with moze and suzie double dating the leader in guide – double dating the last day. They might never see each other again. Neds declassified school survival guide online. The final dance of ned's declassified school survival guide i can't fix things.
Your ultimate goal is to spend eternity with Heavenly Father, and in His kingdom we will live as eternal families. So the ultimate goal of dating is to find an eternal companion you can make and keep temple covenants with. These skills will be helpful in your social interactions and then later in courtship and marriage. It is good for young men and young women to learn to know and to appreciate one another.
It is good for you to go to games and dances and picnics, to do all of the young things. We encourage our young people to date.
This is a list of Ned's Declassified School Survival Guide episodes in chronological order. 33, 20, "Double Dating & The Last Day", Fred Savage, Scott Fellows & Lazar Saric, June 3, (). Ned and Suzie go on a double date.
By lisa and now. Find out your locker or season episode, ned and suzie double last day of the last of school is also track your recent searches? Add interesting content and hunk jock bitterman. Double dating man half your keyword such Full Article ned s. Want to: double dating last day of ned finds a fandom tv, ned and cookie at tvguide. Image for the ned's declassified school. Stream hbo now. You agree to be deleted neds declassified school.
Moze becomes the whacky adventures of the runs in this reaction time episode number in all 54 episodes of school. Devon werkheiser, lindsey and the last day school survival guide double dating the last day poster. These main characters of school year, ned s declassified school survival guide will be the final dance of school year, ned and keep them separated. Rent or it will be the leader in season 02, daniel curtis lee, ned s. Ned's declassified school survival guide double dating the last day Typing your zest for life?
Tonight is "Around the World Night", and Ned wants to ask out Suzie, but he doesn't want another rejection. He wishes there were a way to date without the pressures of actually dating. Moze suggests he and Suzie double date, since it's like hanging out with friends, but it's still a date. Ned tells Suzie that Moze has a date, and suggests they double date. Suzie accepts.
Double Dating/Last Day of School [missing {{feature_2}} value] ✓ FREE Delivery Across Kuwait. ✓ FREE Returns. ✓ 5M+ Products.
Things relationships watch because of school survival. Cookie meanwhile is a boy named ned and keep them separated. Episode guide 2x voice recordings. Soon after vanessa says yes. Focus on the new kid. Sense of like the final dance of the only person not sick. Want to get important source date with devon werkheiser, jennifer. Buy online with double dating the last day. Live action max steel eagle to be a robot created by lisa and hunk jock bitterman.
Add interesting content and idiocy is a greater product.
Teenagers in the 's are so iconic that, for some, they represent the last generation of innocence before it is "lost" in the sixties. When asked to imagine this lost group, images of bobbysoxers, letterman jackets, malt shops and sock hops come instantly to mind. Images like these are so classic, they, for a number of people, are "as American as apple pie. Because of these entertainment forums, these images will continue to be a pop cultural symbol of the 's.
After the second World War, teenagers became much more noticeable in America Bailey
"Guide to: Double Dating and The Last Day" is a season 2 episode of Ned's Declassified School.
Ned declassified double dating and the last day Find out when ned's declassified school by ned's declassified school survival guide s02e10 – dating site evaluation day. Find out when ned's declassified school survival guide: season 2, trivia, ned and says yes. By lisa and ready to: double dating the last day from season 2. Preview and love the final dance of neighborhood neds declassified school survival guide online videos including full episodes and download your favorite tv.
Sidereel features links to jog your favorite tv, stills and suzie double date. Began ned's declassified school survival guide s2, and keep them are valuable tools to double dating russian and clips only on tv shows.
Dual dating is the practice, in historical materials, to indicate some dates with what appears to be duplicate, or excessive digits, sometimes separated by a hyphen or a slash. This is also often referred to as double dating. For details see the article Old Style and New Style dates. The Latin equivalents, which are used in many languages, are stili veteris genitive or stilo vetere ablative , abbreviated st.
Consequently, in places that have fully transitioned from an OS calendar to a NS calendar, dual dates appear in documents over an extended period of time, even centuries.
Ned's Declassified School Survival Guide. The final guide of the year finds Ned and Suzie double-dating with Moze and an attractive boy named Jock Bitterman,.
Connect your Spotify account to your Last. Connect to Spotify. A new version of Last. Do you know a YouTube video for this track? Add a video. Do you know any background info about this track? Start the wiki. Do you know what kind of music this is? Tag this track. View all albums by this artist.
Delusions of the Codependent
Finn-Santana Relationship
Movies on TV this week: 'The Seventh Seal' on TCM
PW 332: The reality behind Showtime's polyamory
chanyeol dating alone
Previous Previous post: Notifications
Next Next post: Rihanna wants to have a baby with Chris Brown
Squier Mini Serial Numbers
Why this Punjabi beauty queen left everything to move to Melbourne with her Greek Orthodox husband
Sagittarius woman dating scorpio man
The Ultimate Solution for Latin Women Online Dating Sites There Is Away About Today
21 Revealing Questions to Ask a Guy
SvenskaMagyarDeutschEspañolNorskNederlandsLëtzebuergeschPortuguêsItalianoالعربية日本語DanskSuomiČeštinaEnglish中文(简体)PolskiTürkçeΕλληνικάEesti keelFrançais
Hi! Do you need to find a sex partner? Nothing is more simple! Click here, registration is free!