%
% 194
%
\chapter{Linear Differential Equations}

\Section{10}{1}{Linear Differential Equations\footnote{The analysis contained in this chapter is mainly theoretical; it
    consists, for the most part, of existence theorems. It is assumed that
    the reader has some knowledge of practical methods of solving
    differential equations; these methods are given in works exclusively
    devoted to the subject, such as
    Forsyth, A Treatise on Differential Equations (1914). %TODO:formatting
  }.
  Ordinary points and singular points.}

In some of the later chapters of this work, we shall be concerned with
the investigation of extensive and important classes of functions
which satisfy linear differential equations of the second order.
Accordingly, it is desirable that we should now establish some general
results concerning solutions of such differential equations.

The standard form of the linear differential equation of the second
order will be taken to be
\begin{equation}
  \frac{\dmeasure^{2} u}{\dmeasure z^{2}}
  + p(z) \frac{\dmeasure u}{\dmeasure z}
  + q(z) u
  = 0
\end{equation}
and it will be assumed that there is a domain $S$ in which both
$p(z), q(z)$ are analytic except at a finite number of poles.

Any point of $S$ at which $p(z), q(z)$ are both analytic will be called
an \emph{ordinary point} of the equation; other points of $S$ will be called
\emph{singular points}.

\Section{10}{2}{Solution\footnote{This method is applicable only to equations of the second order.
    For a method applicable to equations of any order, see
    Forsyth, Theory of Differential Equations, iv. (1902), Ch. I. %TODO:formatting
    } of a differential equation valid in the vicinity of an ordinary point.}

Let $b$ be an ordinary point of the differential equation, and let $S_{b}$ be
the domain formed by a circle of radius $r_{b}$, whose centre is $b$, and its
interior, the radius of the circle being such that every point of $S_{b}$ is
a point of $S$, and is an ordinary point of the equation.

Let $z$ he a variable point of $S_{b}$.

In the equation write
$u = v \exp \thebrace{
  - \half \int_{b}^{z} p(\zeta) \dmeasure \zeta
}$,
and it becomes
\begin{equation}
  \frac{\dmeasure^{2} v}{\dmeasure z^{2}}
  + J(z) v
  = 0
\end{equation}
where
$$
J(z)
=
q(z)
- \half \frac{\dmeasure p}{\dmeasure z}
- \frac{1}{4} \thebrace{p(z)}^{2}.
$$
%
% 195
%

It is easily seen (\hardsubsectionref{5}{2}{2}) that an ordinary point of equation (A) %TODO:ref
is also an ordinary point of equation (B). %TODO:ref

Now consider the sequence of functions $v_{n}(z)$, analytic in $S_{b}$,
defined by the equations
\begin{align*}
  v_{0}(z)
  =& a_{0} + a_{1} (z-b),
  \\
  v_{n}(z)
  =&
  \int_{b}^{z} (\zeta - z) J(\zeta) v_{n-1}(\zeta) \dmeasure \zeta,
  \quad
  (n=1,2,3,\ldots)
\end{align*}
where $a_{0}, a_{1}$ are arbitrary constants.

Let $M, \mu$ be the upper bounds of $\absval{J(z)}$ and $\absval{v_{0}(z)}$ in the domain
$S_{b}$. \emph{Then at all points of this domain}
$$
\absval{v_{n}(z)}
\leq
\mu M^{n} \absval{z-b}^{2n} / (n!).
$$

For this inequality is true when $n=0$; if it is true when $n=0,1,\ldots,m-1$,
we have, by taking the path of integration to be a straight
line,
\begin{align*}
  \absval{v_{m}(z)}
  &=
  \absval{
    \int_{b}^{z} (\zeta-z) J(\zeta) v_{m-1}(\zeta) \dmeasure \zeta
  }
  \\
  &\leq
  \frac{1}{(m-1)!}
  \int_{b}^{z}
  \absval{\zeta-z} \cdot \absval{J(\zeta)}
  \mu M^{m-1}
  \absval{\zeta-b}^{2m-2} \cdot \absval{\dmeasure \zeta}
  \\
  &\leq
  \frac{1}{(m-1)!}
  \mu M^{m} \absval{z-b}
  \int_{0}^{\absval{z-b}} t^{2m-2} \dmeasure t
  \\
  &<
  \frac{1}{m!} \mu M^{m} \absval{z - b}^{2m}.
\end{align*}

Therefore, by induction, the inequality holds for all values of $n$.

Also, since $\absval{v_{n}(z)} \leq \frac{\mu M^{n} r_{b}^{2n}}{n!}$ when $z$ is in $S_{b}$, and
$\sum_{n=0}^{\infty} \mu M^{n} r_{b}^{2n}/(n!)$ converges,
it follows (\hardsubsectionref{3}{3}{4}) that
$v(z) = sum_{n=0}^{\infty} v_{n}(z)$ is a series of analytic functions
uniformly convergent in $S_{b}$; while, from the definition of $v_{n}(z)$,
\begin{align*}
  \frac{\dmeasure}{\dmeasure z} v_{n}(z)
  &= -\int_{b}^{z} J(\zeta) v_{n-1}(\zeta) \dmeasure \zeta,
  \quad (n = 1,2,3,\ldots)
  \\
  \frac{\dmeasure^{2}}{\dmeasure z^{2}} v_{n}(z)
  &= -J(z) v_{n-1}(z);
\end{align*}
hence it follows (\hardsectionref{5}{3}) that
\begin{align*}
  \frac{\dmeasure^{2} v(z)}{\dmeasure z^{2}}
  =&
  \frac{\dmeasure^{2} v_{0}(z)}{\dmeasure z^{2}}
  +
  \sum_{n=1}^{\infty} \frac{\dmeasure^{2} v_{n}(z)}{\dmeasure z^{2}}
  \\
  =&
  - J(z) v(z).
\end{align*}

\emph{Therefore $v(z)$ is a function of $z$, analytic in $S_{b}$, which satisfies
the differential equation
$$
\frac{\dmeasure^{2} v(z)}{\dmeasure z^{2}} + J(z) v(z) = 0,
$$
%
% 196
%
and, from the value obtained for
$\frac{\dmeasure}{\dmeasure z} v_{n}(z)$, it is evident that
$$
v(b) = a_{0},
\quad
v'(b) = \thebrace{\frac{\dmeasure}{\dmeasure z}v(z)}_{z=b} = a_{1},
$$
where $a_{0}, a_{1}$ are arbitrary.}

\Subsection{10}{2}{1}{Uniqueness of the solution.}

If there were two analytic solutions of the equation for $v$,
say $v_{1}(z)$ and $v_{2}(z)$ such that
$v_{1}(b) = v_{2}(b) = a_{0}, v_{1}'(b) = v_{2}'(b) = a_{1}$, then,
writing $w(z) = v_{1}(z) - v_{2}(z)$, we should have
$$
\frac{\dmeasure^{2} w(z)}{\dmeasure z^{2}} + J(z) w(z) = 0.
$$

Differentiating this equation $n-2$ times and putting $z = b$, we get
$$
w^{(n)}(b) + J(b) w^{(n-2)}(b) + TODO
+ \cdots + J^{(n-2)}(b) w(b) = 0.
$$
Putting $n = 2, 3, 4, \ldots$ in succession, we see that all the
differential coefficients of $w(z)$ vanish at $b$; and so, by Taylor's
theorem, $w(z) = 0$; that is to say the two solutions
$v_{1}(z), v_{2}(z)$ are identical.

Writing
$$
u(z) = v(z) \exp \thebrace{
  - \half \int_{b}^{z} p(\zeta) \dmeasure \zeta
},
$$
we infer without difficulty that $u(z)$ is the only analytic solution
of (A) %TODO:ref
such that $u(b) = A_{0}, u'(b) = A_{1}$, where
$$
A_{0} = a_{0},
\quad
A_{1} = a_{1} - \half p(b) a_{0}.
$$

Now that we know that a solution of (A) %TODO:ref
exists which is analytic in $S_{b}$,
and such that $u(b), u'(b)$ have the arbitrary values
$A_{0}, A_{1}$, the simplest method of obtaining the
solution in the form of a Taylor's series is to assume
$u(z) = \sum_{n=0}^{\infty} A_{n} (z-b)^{n}$, substitute this
series in the differential equation and
equate coefficients of successive powers of $z-b$ to zero
(\hardsubsectionref{3}{7}{3}) to
determine in order the values of
$A_{2}, A_{3}, \ldots$ in terms of $A_{0}, A_{1}$.

%\begin{smallfont}
[Note. In practice, in carrying out this process of substitution, the
re;vder will find it much more simple to have the equation 'cleared of
fractions' rather than in the canonical form (A) %TODO:ref
of \hardsectionref{10}{1}. Thus the
equations in examples 1 and 2 %TODO:ref
below should be treated in the form in
which they stand; the factors
$1 - z^{2}, (z-2)(z-3)$
should \emph{not} be
divided out. The same remark applies to the examples of
\hardsectionref{10}{3}, \hardsubsectionref{10}{3}{2}.] %TODO:multiref
%\end{smallfont}

From the general theory of analytic continuation (\hardsectionref{5}{5}) it follows
that the solution obtained is analytic at all points of $S$ except at
singularities of the differential equation. The solution however is
\emph{not}, in general, 'analytic throughout $S$'
(\hardsectionref{5}{2} cor. 2, footnote), %TODO:refs
except at these points, as it may not be one-valued; i.e. it may not
return to the same value when $z$ describes a circuit surrounding one or
more singularities of the equation.
%
% 197
%

[The property that the solution of a linear differential equation is
analytic except at singularities of the coefficients of the equation
is common to linear equations of all orders.]

When two particular solutions of an equation of the second order are
not constant multiples of each other, they are said to form a
\emph{fundamental system}.

\begin{wandwexample}
Shew that the equation
$$
(1 - z^{2}) u'' - 2 z u' + \frac{3}{4} u = 0
$$
has the fundamental system of solutions
\begin{align*}
  TODO
\end{align*}

Determine the general coefficient in each series, and shew that the
radius of convergence of each series is $1$.
\end{wandwexample}
\begin{wandwexample}
Discuss the equation
$$
(z-2)(z-3) u'' - (2z-5) u' + 2u = 0
$$
in a manner similar to that of example 1.
\end{wandwexample}
%
\Section{10}{3}{Points which are regular for a differential equation.}
Suppose that a point $c$ of $S$ is such that, although $p(z)$ or $q(z)$ or
both have poles at $c$, the poles are of such orders that
$(z - c) p(z)$, $(z- c)^{2} q(z)$ are analytic at $c$.
Such a point is called a \emph{regular point}\footnote{The name
  `regular point' is due to
  Thome, Journal fiir Math. lxxv. (1873), p. 266. %TODO:ref
  Fuchs had previously used the phrase ' point of determinateness.'}
for the differential equation. Any poles of $p(z)$ or of $q(z)$ which are
not of this nature are called \emph{irregular points}. The reason for making
the distinction will become apparent in the course of this section.

If $c$ be a regular point, the equation may be written\footnote{Frobenius
  calls this the normal form of the equation.}
$$
(z-c)^{2} \frac{\dmeasure^{2} u}{\dmeasure z^{2}}
+ (z-c) P(z-c) \frac{\dmeasure u}{\dmeasure z}
+ Q(z-c) u = 0,
$$
where $P(z - c), Q(z - c)$ are analytic at $c$; hence, by Taylor's
theorem,
\begin{align*}
  P(z-c) =& p_{0} + p_{1} (z-c) + p_{2} (z-c)^{2} + \cdots, \\
  Q(z-c) =& q_{0} + q_{1} (z-c) + q_{2} (z-c)^{2} + \cdots,
\end{align*}
where $p_{0}, p_{1}, \ldots, q_{0}, q_{1}, \ldots$ are constants; and these series
converge in the domain $S_{c}$. formed by a circle of radius $r$ (centre $c$)
and its interior, where $r$ is so small that $c$ is the only singular
point of the equation which is in $S_{c}$.

Let us assume as a \emph{formal} solution of the equation
$$
u
=
(z-c)^{\alpha}
\thebracket{
  1
  +
  \sum_{n=1}^{\infty} a_{n} (z-c)^{n}
},
$$
where $\alpha, a_{1}, a_{2}, \ldots$ are constants to be determined.
%
% 198
%

Substituting in the differential equation (assuming that the
term-by-term differentiations and multiplications of series are
legitimate) we get
\begin{align*}
  TODO
\end{align*}

Substituting the series for $P(z - c), Q$z - c)$, multiplying out and
equating to zero the coefficients of successive powers of $z - c$, we
obtain the following sequence of equations :

oi- + h-l)a + qo = 0,

, (a + iy + (po-'i-)(o( + l) + qo + ap, + q, = 0, a, (a + 2y + (po -
1) ( + 2 ) + o] + a, (a + l)p, + q + ap, + q, = 0,

an (a -1- nf + (p,-l) ci-]- n) + qo]

+ 2 Qn-m ( + n - m) p,n + ?, + dpn + qn = 0. m=l

The first of these equations, called the indicial equation*,
determines two values (which may, however, be equal) for a. The reader
will easily convince himself that if c had been an irregular point,
the indicial equation would have been (at most) of the first degi'ee;
and he will now appreciate the distinction made between regular and
irregular singular points.

Let a = pi, a = p.2 be the roots i* of the indicial equation F(a) = a?
+ po-l)oi + q, = 0; then the succeeding equations (when a has been
chosen) determine a, a, ..., in order, uniquely, provided that F a +
n) does not vanish when ?i = 1, 2, 3, ...; that is to say, if a = pi,
that p is not one of the numbers TODO; and, if a =
p.., that p is not one of the numbers p - - 1, p., + 2,

Hence, if the difference of the exponents is not zero, or an integer,
it is always possible to obtain two distinct series which formally
satisfy the equation.

Example. Shew that, if ra is not zero or an integer, the equation

is formally satisfied by two series whose leading terms are

.*-' jl+.

-+

.:,

16(H-to) 7' " i ' 16(l-m)

determine the coefficient of the general term in each series, and shew
that the series converge for all values of z.

* The name is due to Cayley, Quarterly Journal, xxi. (1886), p. 326.

t The roots pi, po of the indicial equation are called the expotients
of the differential equation at the point c.

%
% 199
%

\Subsection{10}{3}{1}{Convergence of the expansion o/§ lO'S.}

If the exponents TODO, p are not equal, let p- be that one whose real
part is not inferior to the real part of the other, and let
TODO
then
TODO.
Now, by \hardsubsectionref{5}{2}{3}, we can find a positive number M
such that \ pn\ < Mr-\ \ qn\ < Mr-'\ p.j n + qn i < Mr-'\ where M is
independent of n; it is convenient to take if 1. Taking a = pi, we
see that

aAJMl±,<

M

<

M

\ F p, + ) \ r s+1 since |5 + 1 1 >1.

If now we assume 1 a,,, I < i/"r-" when n=\,2, ...m- 1, we get

ttm. -

2 a -t (pi + vi-t)pi-it qt] + piPm + qr.

F(p, + m)

t I a, t \ piPt + qt] + \ piPm + q,a ! + s (m - 1 - I \ vt

m- 1

m\ 8 +in\

m,M' r-"'+ \ 2 m-t)\ M' r

']

m" 1 1 + smr |

Since 1 1 + snr j 1, because R (s) is not negative, we get

m + 1

a, <

2m

jpn,.-m < j,/' r-'",

and so, by induction, | a i < M' r'" for all values of ?2.

If the values of the coefficients corresponding to the exponent po be
a (iz, ... we should obtain, by a similar induction,

\ an\ < M'Wr-'\

where k is the upper bound of |1- s|~\ \ l - s\~\ 1 1 - |-s |~S ...;
this bound exists when s is not a positive integer.

We have thus obtained two formal series

Wi (z) = z - C)P'

 2 (2 ) = - CY

1+ S aniz-cr

M = 1

1+ ian(z-cY

The first, however, is a uniformly convergent series of analytic
functions when \ z-c\ < rM~\ as is also the second when \ z-c\ < rM-'
K-\ provided

%
% 200
%

in each case that arg (z - c) is restricted in such a way that the
series are one-valued; consequently, the formal substitution of these
series into the left-hand side of the differential equation is
justified, and each of the series is a solution of the equation;
provided always that pi - p., is not a positive integer or zero*.

With this exception, we have therefore obtained a fundamental S3 stem
of solutions valid in the vicinity of a regular singular point. And by
the theory of analytic continuation, we see that if all the
singularities in S of the equation are regular points, each member of
a pair of fundamental solutions is analytic at all points of S except
at the singularities of the equation, which are branch- points of the
solution.

\Subsection{10}{3}{2}{Derivation of a second solution in the case when
  the difference of the exponents is an integer or zero.}

In the case when p - po, = s is a positive integer or zero, the
solution Wo z) found in § lO'Sl may break downf or coincide with Wi
z).

If we write u = w z), the equation to determine f is

(z - cy -r + \ 2 z - cf '-- - + (z-c)Piz-c) = 0, . dz- [ u ( z) ) dz

of which the general solution is

 = A-rB\ ~ - - ry- exp - / - - dz. dz

J W, Z)Y i J 2-C ]

= A+B

(z-c)-P°

= A+Bj (z- c)-P - p g (z) dz,

where A, B are arbitrary constants and g z) is analytic throughout the
interior of any circle whose centre is c, which does not contain any
singu- larities of P ( - c) or singularities or zeros of z - c)"''i w
z); also g (c) = 1.

00

Let g z)= + S gn.(z-cy\

n = l

Then, Hsi O,

 =A+Br\ l+i(z-cf ( z-c)-'-'dz

J [ w = l J

= A+B --(z- c)- - 'i - (z - cy-' + gs log (z - c)

\ S n=l n

+ % - (z - cY-' n = s+l n -s

* If Pi -p2 is a positive integer, k does not exist; if p = p2, the
two solutions are the same.

t The coefficient a/ may be indeterminate or it may be infinite; in
the former case to-, (z) will be a solution containing two arbitrary
constants qq' and aj; the series of which Ug is a factor will be a
constant multiple of wi (z).

%
% 201
%

Therefore the general solution of the differential equation, which is
analytic at all points of C (c excepted), is

Awi z) + B [gsti\ (z) log (z-c) + w (z)], where, by \hardsubsectionref{2}{5}{3}, w (z) = z
- c) p- \ - - + S h,, z - c)"l,

the coefficients hn being constants.

When s = 0, the corresponding form of the solution is

Aw,(z) + B w, (z) log (z-c) + (z- cY X hn (z - c)"

L n = l \

The statement made at the end of \hardsubsectionref{10}{3}{1} is now seen to hold in the
exceptional case Avhen s is zenj or a positive integer.

In the special case when gg = 0, the second solution does not involve
a logarithm.

The solutions obtained, which are valid in the vicinity of a regular
point of the equation, are called regular integrals.

Integrals of an equation valid near a regular point c may be obtained
practically by first obtaining w- z), and then determining the
coefficients in

30

a function u\ (z) = S hn z - c)''-+'*, by substituting lu (z) log (z -
c) + n\ z) in

the left-hand side of the equation and equating to zero the
coefficients of the various powers of z - c in the resulting
expression. An alternative method due to Frobenius* is given by
Forsyth, Treatise on Differential Equations, pp. 243-258.

Example L Shew that integrals of the equation

<Pu \ du -

regular near z - are

Wi z) = l+ 2

and ..,(.) log.- 2 . ( \ + \ +...+-j.

Verify that these series converge for all vahies of z. Example 2. Shew
that integrals of the equation

,d u, -.sdu 1

regular near 2 = are

'°'W = + .!.( 2.4...2,. )-

and .,(.)log. + 4 j( \ - - j ( \ \ + \ \ ...\ \ J..

Verify that these series converge when | . i < 1 and obtain integrals
regular near 3= L * Journal fur Math, lxxvi. (1874), pp. 214-224.

%
% 202
%

Example 3. Shew that the hypergeometric equation

z z) :i + c-(a->rh + ) z -ahu = Q dz dz

is satisfied by the hypergeometric series of \hardsubsectionref{2}{3}{8}.

Obtain the complete sohition of the equation when c = .

\Section{10}{4}{Solutions valid for large values of $z$.}

Let = l/5i; then a solution of the differential equation is said to be
valid for ' large values of | | ' if it is valid for sufficiently
small values of Ui |; and it is said that ' the point at infinity is
an brdinary (or regular or irregular) point of the equation ' when the
point j = is an ordinary (or regular or irregular) point of the
equation when it has been transformed so that Zi is the independent
variable.

Since

we see that the conditions that the point 2= oc should be (i) an
ordinary point, (ii) a regular point, are (i) that 2z - z~p 2), z*q z)
should be analytic at infinity \hardsubsectionref{5}{6}{2}) and (ii) that zp z), z-q (z)
should be analytic at infinity.

Example 1. Shew that every point (including infinity) is either an
ordinary point or a regular point for each of the equations

z l-z)-jj ->r c- a-vh + ) z - -ahu = 0,

(1- 2), - -2s +?i(?i + l) = 0,

where a, \&, c, n are constants.

Exam-pie 2. Shew that every point except infinity is either an
ordinary point or a regular point for the equation

 S-"+4"- ( '- ') =°'

where n is a constant.

Example 3. Shew that the equation

has the two solutions

,, .,, d' u du .

2\ 1 i i 3.4.5.6 1

 " 3' 7 ' 277 7 ' 2. 4. 7. 9 z

the latter converging when | s | > 1.

\Section{10}{5}{Irregular singularities and confluence.}

Near a point which is not a regular point, an equation of the second
order cannot have two regular integrals, for the indicial equation is
at most of the first degree; there may be one regular integral or
there may be none. We shall see later (e.g.\hardsectionref{16}{3}) what is the nature
of the solution near

%
% 203
%

such points in some simple cases. A general investigation of such
solutions* is beyond the scope of this book.

It frequently happens that a differential equation may be derived from
another differential equation by making two or more singularities of
the latter tend to coincidence. Such a limiting process is called
confluence; and the former equation is called a confluent form of the
latter. It will be seen in § lO'G that the singularities of the former
equation may be of a more complicated nature than those of the latter.

\Section{10}{6}{The differential equations of mathematical physics.}

The most general differential equation of the second order which has
every point except a, a, as, a and oo as an ordinary point, these
five points being regular points with exponents a, ySr at ar r = 1,
2, 3, 4) and exponents fjbj, /j,2 at X, may be verified f to be

dz'' Vti z-ar )dz \ rZi z-ayf T (z-ar)

r = \

where A is such that;): / i and i., are the roots of

and B, C are constants.

The remarkable theorem has been proved by Klein§ and B6cher|| that all
the linear differential equations which occur in certain branches of
Mathematical Physics are confluent forms of the special equation of
this type in which the difference of the two exponents at each
singularity is |; a brief investigation of these forms will now be
given.

If we put y9,. = a,. +, (r = l, 2, 8, 4) and write in place of z, the
last written equation becomes

d i U h- Adu (4 ar(ar + h) 1 ? + 2 r+C ] \ Q

r=l

* Some elementary investigations are given in Forsyth's Differential
Equations (1914). Complete investigations are given in his Theory of
Differential Equations, iv. (1902).

t The coefficients of - and u must be rational or they would have an
essential singularity dz

4 4

at some point; the denominators of p z), q z) must be H z-a ), n z-a )
respectively;

r=l J-=l

putting p(z) and q (z) into partial fractions and remembering that p
z) = 0(z- ), q z) - 0(z~-) as 1 2 - 00, we obtain the required result
without difficulty.

4

X It will be observed that mi, fJ-i are connected by the relation
M1+M2+ 2 (a,. + /3 ) = 3.

j=i

§ Ueher lineare Differentialgleichungen der zweiter Ordnung (1894), p.
40; see also Vorlesung

Uber Lamd schen Funktionen.

II Ueber die Reihenentwickelungen der Potentialtheorie (1894), p. 193.

%
% 204
%

where (on account of the condition yu.., - f i = 2)

\ r=l

This differential equation is called the generalised \\Lame\\ equation.

It is evident, on writing i = ao throughout the equation, that the
confluence of the two singularities a, a., yields a singularity at
which the exponents a, /3 are given by the equations

a + /3 = 2 ( ! + oo), a = a, (oc, + h) + a, (a, + |) + D,

where D = (Aa - + 2Ba + C)/ (ai - a. ) (a - a ) .

Therefore the exponent-difference at the confluent singularity is not
, but it may have any assigned value by suitable choice of B and C. In
like manner, by the confluence of three or more singularities, we can
obtain one irregular singularity.

By suitable confluences of the five singularities at our disposal, we
can obtain six types of equations, which may be classified according
to (a) the number of their singularities with exponent-difference |,
(6) the number of their other regular singularities, (c) the number of
their irregular singu- larities, by means of the following scheme,
which is easily seen to be exhaustive*:

(a)

(b)

(c)

(I)

3

1

Lame

(H)

2

1

Mathieu

(III)

1

2

Les:eudre

(IV)

1

1

Bessel

(V)

1

1

Weber, Hermite

(VI)

1

Stokes t

These equations are usually known by the names of the mathematicians
in the last column. Speaking generally, the later an equation comes in
this scheme, the more simple are the properties of its solution. The
solutions of (II)-(YI) are discussed in Chapters xv-xix of this work,
and J of (I) in Chapter xxiii. The derivation of the standard forms of
the equations from the generalised \\Lame\\ equation is indicated by the
following examples :

* For instance the arrangement (a) 3, h) 0, (c) 1 is inadmissible as
it would necessitate six initial singularities.

t The equation of this type was considered by Stokes in his researches
on Diffraction, Camb. Phil. Trans, ix. (1856), pp. 168-182; it is,
however, easily transformed into a particular case of Bessel's
equation (example 6, below).

t For properties of equations of type (I), see the works of Klein and
Forsyth cited at the end of this chapter; also Todhunter, The
Functions of Laplace, \\Lame\\ and Bessel (1875).

%
% 205
%

Example . Obtain \Lame's equation

r=l

(where h and n are constants) by taking

n, =0 = 03 = 04 = 0, 8B = n('/i- l) a, 4(7=/i 4, and making 04 - x .

Example 2. Obtain the equation

dc' \ rc- y dc 4c (c- 1) ~ '

(where a and j are constants) by taking ai = 0, 0-2 = 1, and making 3
= c/4 -x. Derive Mathieu's equation \hardsectionref{19}{1})

-7 + (a + 16jcos 2i) =

by the substitution f = cos 2.

Example 3. Obtain the equation

 *4./K\ JLW, 1 f!L( L)\ \ !!!!\ l \ \ n

c r " (r c- 1/ c/c * I f f - ij c (c- 1) ~ '

by taking

 i = 2=l, 3 = a4 = 0, ai = a.j = a3 = 0, a4 = |.

Derive Legendre's equation (§§ 15-13, 15'5)

by the substitution ( = z~\

Example 4. By taking, = (/.2 = 0, 01 = 02 = 03 = 04 = 0, and making
a3 = a4-*-x, obtain the equation

Derive Bessel's equation (\hardsubsectionref{17}{1}{1})

d u

dz " dz

gd u du,,

by the substitution C=z' .

Example 5. By taking i=0, 01 = 02 = 03 = 04 = 0, and making (/o = 3 =
a4-*-x, obtain the equation

. dhi du,,,,,

f +i5 +i( +*-if) =o.

Derive Weber's equation \hardsectionref{16}{5})

d u,,, x

  + ( +*- -') =o

by the substitution f= 2.

Example 6. By taking 0 =0, and making 0 -9- x ( = 1, 2, 3, 4), obtain
the equation

  + ( if+Ci) = 0. By taking

u = B,C+C\ r-v, B,C+C\ = IB,z), shew that

 d v dv,,,.

%
% 206
%

Example 7. Shew that the general form of the generalised \\Lame\\ equation
is un- altered (i) by any homographic change of independent variable
such that qo is a singular point of the transformed equation, (ii) by
any change of dependent variable of the type

'll = z-ar) V.

Example 8. Deduce from example 7 that the various confluent forms of
the generalised \Lame\ equation may always be reduced to the forms given
in examples 1-6.

[Note that a suitable homographic change of variable will transform
any three distinct points into the points 0, 1, qo .]

\Section{10}{7}{Linear differential equations ivith three singularities.}

d?u, du, . \ Let d ' dz' '

have three, and only three singularities*, a, h, c; let these points
be regular points, the exponents thereat being a, a!; /3, /3'; 7,
7'.

Then p z) is a rational function with simple poles at a, b, c, its
residues at these poles being 1 - a - a', 1 - /3 - /S', 1 - 7 - 7';
and as 2 co, p z)- 2z- is z- ). Therefore

  z - a z - z - c

andf a + a' + + y8' + 7 + 7' = 1.

In a similar manner

,, [aoL a-h) a-c) B 'ih-c) h-a) yy' (c - a) c - b) l z - a z - b z - c

1

X

 z - a) z- b) z - g) and hence the differential equation is

d-'u U- o- ' 1-/3-/3 1-7-7 ) du

dz' [ z - a z - b z - c \ dz

  oia' a-b ) (a-c) /3 ' (b -c)(b-a) 77 (c -a)(c- b) \ z - a z - b z~c
I

= 0.

 z- a) z - b) z- c) This equation was first given by Papperitzj,

To express the fact that u satisfies an equation of this type (which
will be called Riemann's P-equation), Riemann§ wrote

fa b c \ u = p a /3 7 z[. W /3' 7 J

* The point at infinity is to be an ordinary point.

t This relation must be satisfied by the exponents.

+ Math. Ann. xxv. (1885), p. 21B.

§ Abh. d. k. Ges. d. Wiss. zu Gottingen, vii. (1857). It wiU be seen
from this memoir that, although Kiemann did not apparently construct
the equation, he must have inferred its existence from the
bypergeometric equation.

%
% 207
%

The singular points of the equation are placed in the first row with
the corresponding exponents directly beneath them, and the independent
variable is placed in the fourth column.

Example. Shew that the hypergeometric equation

d-u,, .,, . c?

,,, d-u,, 7,, . du,

is defined by the scheme

j' 00 1 \

pJ a 2-.

[l-o h c - a-h j

10 71. Transformations of Riemanns P-equation. The two transformations
which are typified by the equations

a b

z-h

(I) iv l) [v k) z\ = P a -k -k-l y- l

; (

b

c

7 W

y3

7

a'

/3'

i

a

b

c

(II) P<a

a'

7

IS'

7'

c

a' + k '-k-l y'+l

J (a /5' 7' J

(where z, a, 61, Cj are derived from z, a, b, c by the same
homographic transformation) are of great importance. They may be
derived by direct transformation of the differential equation of
Papperitz and Riemann by suitable changes in the dependent and
independent variables respectively; but the truth of the results of
the transformations may be seen intuitively when we consider that
Riemann's P-equation is determined uniquely by a knowledge of the
three singularities and their exponents, and (I) that if
$$
TODO
$$
IS'
$$
TODO
$$
then Ui = i- -jj (-37) satisfies a differential equation of the second

order with the same three singular points and exponents a + k, a +k;
l3 - k - l,j3' - k - l; y+ I, y' + I; and that the sum of the
exponents is 1,

Az + B Also (II) if we write z = j -, the equation in z- is a linear
equation

of the second order with singularities at the points derived from a,
b, c by this homographic transformation, and exponents a, a; fS, ';
y, y thereat.

%
% 208
%

\Subsection{10}{7}{2}{The connexion of Riemanns P-equation with the hypergeometric equation.}

B) means of the results of § lO'Tl it follows that

 a - a /3 + a + y 7' - 7 J

where x =

a' - a /3' + a + 7 7' - 7 (z - a)(c- b)

(z - h) c - a)'

Hence, by \hardsectionref{10}{7} example, the solution of Riemann's P-equation can

always be obtained in terms of the solution of the hypergeometric
equation

1 1,7 r n,' -1, z - a) c- ))

whose elements a, 0, c, x are a+ + 7, a-t-p+7, i+a - a, j - 7 -

 z-b) c - a)

respectively.

\Section{10}{8}{Linear differential equations tuith two singularities.}

If, in \hardsectionref{10}{7}, we make the point c a regular point, we must have

, ',, aa(a-b)(a-c) /3/3' (b - c) (b - a) l\ \ y=0, 77' = and - 4. 1
\ Lv 1 j g

divisible by - c, in order that p (z) and q (z) may be analytic at c.
Hence a + a + + jS' = 0, aa = yS/S', and the equation is d-ii j 1 - a
- a' 1 + a + a'] du atx a - by u dz- \ z - a z - b ) dz z- of z - b)-
'

of which the solution is

. fz - aY -D fz-( u=A J + B

\ z -b) \ z - bj

that is to say, the solution involves elementary functions only. When
a - a, the solution is

. fz-aY fz - aY fz-a

bJ'

REFEREXCES.

L. FuCHS, Journal fur Math. Lxvi. (1866), pp. 121-160.

L. W. Thomis, Journal fur Math. Lxxv. (1873), pp. 265-291, Lxxxvii.
(1879), pp. 222-349.

L. ScHLESiNGER, Handbuch der linearen Differentialgleichungen.
(Leipzig, 1895-1898.)

G. Frobenius, Journal fur Math. Lxxvi. (1874), pp. 214-235.

G. F. B. Riemann, Ges. Math. Werke, pp. 67-87.

F. C. Kleix, Ueber Uneai-e Differentialgleichungen der zineiter
Ordnung. (Gottingen, 1894.)

A. R. Forsyth, Theory of Differential Equations, iv. (1902).

T. Craig, Differential Equations. (New York, 1889.)

E. Goursat, Coxirs d\ inalyse, 11. (Paris, 1911.)

%
% 209
%

Miscellaneous Examples.

1. Shew that two solutions of the equation

are 2 - 3 7 2* + ..., 1- 2 + ..., and investigate the region of
convergence of these series.

2. Obtain integrals of the equation

d" 1,, ON

regular near 2 = 0, in the form

. i = "r + r6 + 1024 + -/'

3

 2 = Wll0g2-jg+....

3. Shew that the equation

has the solutions

TODO

and that these series converge for all values of z.

4. Shew that the equation

dZ tr=l Z-ttr ) dz \ r=i z - a,.Y r=l*- rj

where

2 (a, + 3,) = -2, 2 Z), = 0, 2 (a, A-+ari3r) = 0, 2 (a,2/) + 2a,a,i3,)
= 0,

r=I r=I r=l r=l

is the most general equation for which all points (including x ),
except aj, a-.,, ... a, are ordinary points, and the points a are
regular points with exponents a, r respectively.

\addexamplecitation{Klein.}

5. Shew that, if /iJ + y + /3' + y' =, then

(0 X 1 I i-1 X 1 ~|

P\ 0 y 22' p' 2/3 y 2-. \addexamplecitation{Riemann.}

h \& y J i y' 23' y J

[The dififerential equation in each case is

c/22 + 2- 1 dz r 22 . IJ,2\ l ""J

6. Shew that, if y + y' = and if co, or are the complex cube roots of
unity, tfeen rO X 1 \ rl CO 0)2 j

pJo y 23- = p'y y y 2-. \addexamplecitation{Riemann.}

U i y J ly y 7' >'

[The differential equation in each case is

d' u 222 du yy zu,

C£22 23\ l dz z -\ f -

W. M. A. 1

%
% 210
%

7. Shew that the equation

(l\ .2)g\ (2o + i)2 Vn( + 2a) =

is defined b\ \ the scheme

pj -n z

\ jf-a n + 2a - a j and that the equation

may be obtained from it by taking a = l and changing the independent
variable.

8. Discuss the solutions of the equation

dhi,,, dii (,1

\addexamplecitation{Halm.}

'-U,, -.du (, . 1 \ + ( + l+m) + (/i + l+ mj w =

valid near s = and those valid near s = oo . \addexamplecitation{Cunningham.}

 22

9. Discuss the solutions of the equation

valid near 2=0 and those valid near z=<x, . Consider the following
special cases :

(i) pi= -% (ii) /i - 5, (iii) /x + i/ = 3.
\addexamplecitation{Curzon.}

10. Prove that the equation '

s(l-2) +-(l-22) + (a2 + fe)w =

has two particular integrals the product of which is a single-valued
transcendental function. Under what circumstances are these two
pax'ticular integrals coincident ? If their product be F z), prove
that the particular integrals are

where C is a determinate constant.
\addexamplecitation{Lindemann; see \hardsectionref{19}{5}.}

l. Prove that the general linear differential equation of the third
order, whose singularities are 0, 1, oc, which has all its integrals
regular near each singularity (the exponents at each singularity being
1, 1, - 1), is

dhi (2 2 \ d \ \ \ 3 1 \ "1 du

d W ) dz V z z-\ y z- If] dz \ \ 3cos2a 3sin2a 1 1 \ n

+ p - w ) " ¥ ' " m '

where a n y ha 'e any constant value. \addexamplecitation{Math. Trip. 1912.}


\subsection{Sequence-to-sequence}
{\color{red}
	Seq2seq turns one sequence into another sequence (sequence transformation). It does so by use of a recurrent neural network (RNN) or more often LSTM or GRU to avoid the problem of vanishing gradient. The context for each item is the output from the previous step. The primary components are one encoder and one decoder network. The encoder turns each item into a corresponding hidden vector containing the item and its context. The decoder reverses the process, turning the vector into an output item, using the previous output as the input context.[2]

	Optimizations include:[2]

	Attention: The input to the decoder is a single vector which stores the entire context. Attention allows the decoder to look at the input sequence selectively.
	Beam Search: Instead of picking the single output (word) as the output, multiple highly probable choices are retained, structured as a tree (using a Softmax on the set of attention scores[6]). Average the encoder states weighted by the attention distribution.[6]
	Bucketing: Variable-length sequences are possible because of padding with 0s, which may be done to both input and output. However, if the sequence length is 100 and the input is just 3 items long, expensive space is wasted. Buckets can be of varying sizes and specify both input and output lengths.
	Training typically uses a cross-entropy loss function, whereby one output is penalized to the extent that the probability of the succeeding output is less than 1.[6]
}


%!TEX root = ms.tex
\section{Performance of the selectors}
\label{sec:simulation}
\subsection{Some other selectors}
In this section we use computer simulations to explore the behavior of different criteria when used for model selection under linear restrictions (variable selection and general linear restrictions). In addition to the criteria already discussed, we also consider two other well-known criteria:
\begin{equation*}
\text{BIC}(k) = n \log\left( \frac{\text{RSS}(k)}{n} \right) +\log(n) k
%\label{eq:bic}
\end{equation*}
\citep{schwarz1978estimating}, and generalized cross-validation (GCV)
\begin{equation}
\text{GCV}(k) = \text{RSS}(k)\frac{n^2}{(n-k)^2}.
\label{eq:gcv_subsetselection}
\end{equation}
BIC is a consistent criterion, in the sense that under some conditions, if the true model is among the candidate models, the probability of selecting the true model approaches one, as the sample size becomes infinite. GCV, derived by \citet{craven1978smoothing} in the context of smoothing, is equivalent to the mean square over degrees of freedom criterion proposed by \citet{tukey1967discussion}. By comparing the expressions of \eqref{eq:gcv_subsetselection} and \eqref{eq:sp_subsetselection}, GCV and S$_p$ only differ by a multiplicative factor of $1 + k / [(n-k)(n-1)]$.

By analogy to the criteria discussed in Section \ref{sec:ic_fixedx} and \ref{sec:ic_randomx}, if we substitute $k=p-m$ into the expressions of BIC$(k)$ and GCV$(k)$, we obtain their corresponding expressions for general linear restrictions BIC$(R,r)$ and GCV$(R,r)$, respectively. We also consider two types of the cross-validation (CV): 10-fold CV (denoted as 10FCV) and leave-one-out CV (LOOCV). The LOOCV is based on the PRESS(k) statistic for the variable selection problem and PRESS(R,r) for the general restriction problem.

\subsection{Random-X}
We first consider the variable selection problem. The candidate models include the predictors of $X$ in a nested fashion, i.e. the candidate model of size $k$ includes the first $k$ columns of $X$ ($X_1,\cdots,X_k$). We describe the simulation settings reported here; description of and results from all other settings (243 configurations in total) can be found in the Online Supplemental Material\footnote{\url{https://github.com/sentian/RAICc}}, where we also provide the code to reproduce all of the simulation results in this paper. The sample sizes considered are $n\in\{40, 1000\}$, with number of predictors $p=n-1$, being close to the sample size. Predictors exhibit moderate correlation with each other of an AR(1) type (see the Online Supplemental material for further details), and the strength of the overall regression is characterized as either low (average $R^2$ on the set of true predictors roughly $20\%$) or high (average $R^2$ on the set of true predictors roughly $90\%$). The true model is either sparse (with six nonzero slopes) or dense (with $p$ nonzero slopes exhibiting diminishing strengths of coefficients; \citealp{Taddy2017}). The design matrix $X$ is random. In each replication, we generate a matrix $X$ such that the rows $x_i$ ($i=1,\cdots,n$) are drawn from a $p$-dimensional multivariate normal distribution with mean zero and covariance matrix $\Sigma_0$, and we draw the response $y$ from the conditional distribution of $y|X$ based on \eqref{eq:truemodel}. The entire process is repeated $1000$ times.

We consider the following metrics to evaluate the fit. The values of each criterion over all of the simulation runs are plotted using side-by-side boxplots, with the average value over the simulation runs given below the boxplot for the corresponding criterion.
\begin{itemize}
  \item Root mean squared error for random-X:
  \begin{equation*}
    \text{RMSER} = \sqrt{ E_{X^n} \lVert X^n\hat\beta-X^n\beta_0 \rVert_2^2 } =  \sqrt{ (\hat{\beta}-\beta_0)^T \Sigma_0 (\hat{\beta}-\beta_0) }.
  \end{equation*}

  \item KL discrepancy for random-X \eqref{eq:KLR} in the log scale (denoted as logKLR).

  \item Size of the subset selected for variable selection problem, and number of restrictions in the selected model for general restriction problem.
\end{itemize}

The results are presented in Figures \ref{fig:subsetselection_randomx_hsnr_largep} and \ref{fig:subsetselection_randomx_lsnr_largep}. We find that RAICc provides the best predictive performance and the sparsest subset while rarely underfitting, compared to other information criteria designed for random-X, including RC$_p$ and S$_p$. The underperformance of RC$_p$ and S$_p$ is due to overfitting. S$_p$, as an estimate of the squared prediction error, is a product of an unlogged residual sum of squares and a penalty term that is increasing in $k$. This results in higher variability in S$_p$ for models that overfit, thereby potentially increasing the chances for spurious minima of these criteria at models that drastically overfit. In RAICc, on the other hand, the residual sum of squares is logged, thereby stabilizing the variance and avoiding the problem. RC$_p$ drastically overfits in all scenarios, reflecting the price of estimating $\sigma_0^2$ using the full model, especially when $p$ is close to $n$. S$_p$, on the other hand, estimates $\sigma_0^2$ using the candidate model, which mitigates the problem. Nevertheless, S$_p$ also can sometimes strongly overfit, but only when $n$ is small. Even for large $n$, S$_p$ selects slightly larger subsets on average than does RAICc.

We also note that information criteria designed for random-X generally perform better than their counterparts for the fixed-X case. Both C$_p$ and FPE are largely outperformed by RC$_p$ and S$_p$, respectively. The advantage of RAICc over AICc is statistically significant in most scenarios, based on the Wilcoxon signed-rank test (the $p$-value for the test comparing the criteria for RAICc and AICc is given above the first two boxes in the first two columns of the table), but is not obvious in a practical sense. The only place that we see an advantage of AICc is for \{Dense model, $n=40$ and high signal\}. In this scenario, a model with many predictors with nonzero slopes can predict well, but that advantage disappears when there is a relatively weak signal, as in that situation the added noise from including predictors with small slopes cannot be overcome by a small error variance.

We further note that choosing the appropriate family of information criteria (the KL-based AICc and RAICc) is more important than choosing the information criteria designed for the underlying designs of $X$. AICc, despite being designed for fixed-X, outperforms RC$_p$ and S$_p$, which are designed for random-X in all of the scenarios, in terms of both predictive performance and providing sparse results. The KL-based criteria have a clear advantage compared to the squared-error based criteria.

Finally, we note some other findings that have been discussed previously in the literature. Despite its apparently strong penalty, BIC often chooses the model using all predictors when $n$ is close to $p$, as discussed in \citet{hurvich1989regression} and \citet{baraud2009gaussian}. We also see that even though GCV has a similar penalty term as S$_p$, it is more likely to suffer from overfitting. Unlike S$_p$, GCV can sometimes drastically overfit even when $n$ is large. The overfitting problem of GCV was also observed in the context of smoothing by \citet{hurvich1998smoothing}. We further find that 10-fold CV performs better than LOOCV, the latter of which sometimes drastically overfits. The tendency of LOOCV to strongly overfit was noted by \citet{scott1987biased} and \citet{hall1991local} in the context of smoothing. \citet{zhang2015cross} showed that when applied as selection rules, the larger validation set used by 10-fold CV can better distinguish the candidate models than can LOOCV, and this results in a model with smaller predictive error. RAICc performs better than 10-fold CV for small $n$, and performs similarly for large $n$. Computationally, 10-fold CV is ten times more expensive compared to RAICc, and since the split of validation samples is random, 10-fold CV can select different subsets if applied multiple times on the same dataset; that is, the result of 10-fold CV is not reproducible. The fact that LOOCV provides better estimate of the test error while being outperformed by 10-fold CV, further emphasizes the difference between the goal of providing the best estimate of the test error, and the goal of selecting the models with the best predictive performance. Clearly, KL-based criteria (AICc and RAICc) bridge the gap between the two goals more effectively than the squared-error based criteria (including cross-validation).

In a related study by \citet{leeb2008evaluation}, the author found that S$_p$ and GCV outperform AICc under random-X, but those results are not directly comparable to ours. That paper did not consider the case where $p$ is extremely close to $n$, which is the scenario that most separates the performances of the different criteria.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx_VS_hsnr.eps}
  \caption{Results of simulations for variable selection. Random-X, high signal and $\rho=0.5$. The Sparse and Dense models correspond to the VS-Ex2 and VS-Ex3 configurations (details are given in the Online Supplemental Material). The first column refers to RMSE, the second column corresponds to KL discrepancy (in log scale), and the third column gives the number of variables in the selected model with nonzero slopes, jittered horizontally and vertically, so the number of models with that number of nonzero slopes can be ascertained more easily. The mean values of the evaluation metrics for each criterion are presented at the bottom of each graph. The p-values of the Wilcoxon signed-rank test (paired and two-sided) for comparing RAICc and AICc are also presented.}
  \label{fig:subsetselection_randomx_hsnr_largep}
\end{figure}

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx_VS_lsnr.eps}
  \caption{Results of simulations for variable selection. Random-X, low signal and $\rho=0.5$. }
  \label{fig:subsetselection_randomx_lsnr_largep}
\end{figure}

We next consider the general restriction problem. We take $\beta_0 = [2,2,2,1,1,1]^T$, $n\in\{10,40\}$, moderate correlations between the predictors, and either high or low signal levels. The candidate models are constructed in the following way. We consider a set of restrictions: $\beta_1=\beta_4$, $\beta_1=2\beta_2$, $\beta_1=\beta_2$, $\beta_2=\beta_3$, $\beta_4=\beta_5$, $\beta_5=\beta_6$, where the last four restrictions hold for our choice of $\beta_0$. We then consider all of the possible subsets of the six restrictions, resulting in $64$ candidate models in total. The detailed configurations and complete results for this and other examples of the general restriction problem ($54$ scenarios in total) are given in the Online Supplemental Material.

We see from Figure \ref{fig:generalrestriction_randomx} that differences in performance between the criteria are less dramatic. This is not surprising, since for these models the number of parameters never approaches the sample size. Still, RAICc is consistently the best selection rule for small sample size $n$, and it is second-best for large $n$, where it is outperformed by BIC (note that BIC has a strong tendency to select too few restrictions when the sample is small, which corresponds to overfitting in the variable selection context). We also note an advantage of RAICc over AICc, with AICc having a stronger tendency to select too few restrictions.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx_GR-Ex1.eps}
  \caption{Results of simulations for general restrictions. Random-X, $\rho=0.5$. The configuration of the model is GR-Ex1 (details can be found in the Online Supplemental Material). Third column gives the number of restrictions in the selected models, jittered horizontally and vertically. }
  \label{fig:generalrestriction_randomx}
\end{figure}

Finally, we extend the general restriction example by including restrictions that force additional predictors to have zero coefficients (as in the variable selection problem). Besides the six restrictions specified, we also consider $\beta_i=0$ for $i=7,\cdots,p$ resulting in $p$ possible restrictions in total. The candidate models are formulated by excluding the restrictions in a nested fashion. We start from the model including all $p$ restrictions (corresponding to the null model), and the next model includes the $p-1$ restrictions except the first one $\beta_1=\beta_4$. The process is repeated until all restrictions are excluded (the full model including all predictors with arbitrary slopes) resulting in $p+1$ candidate models in total. The true coefficient vector is the same as that used in Figure \ref{fig:generalrestriction_randomx}, implying that the correct number of restrictions is $p-2$. We present the detailed configurations and complete results for this and other examples ($243$ scenarios in total) in the Online Supplemental Material.

We see from Figure \ref{fig:subsetgeneral_randomx} that our findings for the variable selection problem also hold in this case. This is not surprising, since variable selection is just a special example of general restrictions, and in this scenario the set of candidate models includes ones where the number of parameters is close to the sample size. Thus, overall, RAICc and AICc are the best performers among all of the selectors. RAICc tends to provide the sparsest subset (or select more restrictions), while rarely underfitting, having a slight advantage over AICc in terms of predictive performance.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx_GR-Ex4.eps}
  \caption{Results of simulations for general restrictions. Random-X, $\rho=0.5$. The configuration of the model is GR-Ex4 (details can be found in the Online Supplemental Material).}
  \label{fig:subsetgeneral_randomx}
\end{figure}

\iffalse
\subsubsection{Without a true model}
\begin{itemize}
	\item Omit: The configuration is the same as Sparse-Ex1, but with the 6-th predictor treated as missing for all the fitting procedures.
	\item Exponential: We take $\Sigma = I$. The responses are generated by $y_i=exp(4i/n) + \epsilon_i$, for $i=1,\cdots,n$, where $\epsilon_i$ are independent $\mathcal{N}(0, \sigma_0^2)$.
\end{itemize}
	We consider a fixed trigonometric configuration of $X$ that is studied by \citet{Hurvich1991}, where $X$ is an $n$ by $p$ matrix with components defined by
$$ x_{t, 2j-1} = \sin\left(\frac{2\pi j}{n}t\right),$$
and
$$ x_{t,2j} = \cos\left(\frac{2\pi j}{n}t\right),$$
for $j=1,\cdots,p/2$ and $t=0,\cdots,n-1$. The responses are generated by $y_i=exp(4i/n) + \epsilon_i$, for $i=1,\cdots,n$, where $\epsilon_i$ are independent $\mathcal{N}(0, \sigma_0^2)$.
\fi

\iffalse
\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx/subset_selection/smallp_hsnr.eps}
  \caption{Random-X, high signal and $\rho=0.5$. The mean values of the evaluation metrics for each criterion are presented at the bottom of each graph. The p-values of the Wilcoxon signed-rank test (paired and two-sided) for comparing RAICc and AICc are also presented.}
  \label{fig:subsetselection_randomx_hsnr_smallp}
\end{figure}

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/randomx/subset_selection/smallp_lsnr.eps}
  \caption{Random-X, low signal and $\rho=0.5$. Other details are the same as in Figure \ref{fig:randomx_hsnr_smallp}.}
  \label{fig:subsetselection_randomx_lsnr_smallp}
\end{figure}
\fi

\subsection{Fixed-X}
The simulation structure for random-X can also be applied to fixed-X. We only generate the design matrix $X$ once and draw $1000$ replications of the response vector $y$ from the conditional distribution of $y|X$ based on \eqref{eq:truemodel}. The evaluation metrics for fixed-X are as follows. The complete simulation results are given in the Online Supplemental Material.
\begin{itemize}
  \item Root mean squared error for fixed-X:
  \begin{equation*}
    \text{RMSEF} = \sqrt{ \frac{1}{n}\lVert X\hat\beta-X\beta_0 \rVert_2^2 }.
  \end{equation*}

  \item KL discrepancy for fixed-X \eqref{eq:KLF} in the log scale (denoted as logKLF).

  \item Size of the subset selected for variable selection problem, and number of restrictions in the selected model for general restriction problem.
\end{itemize}

The patterns for the fixed-X scenario are similar to those for random-X, as can be seen in Figures \ref{fig:subsetselection_fixedx_hsnr_largep}, \ref{fig:subsetselection_fixedx_lsnr_largep}, \ref{fig:generalrestriction_fixedx} and \ref{fig:subsetgeneral_fixedx}. In some ways this is surprising, in that the random-X versions of the criteria still seem to outperform the fixed-X versions, even though that is not the scenario for which they are designed. This seems to be related to the tendency for the fixed-X versions to overfit (or choose too few restrictions) compared to their random-X counterparts, which apparently works against the goal of selecting the candidate with best predictive performance. Otherwise, the KL-based criteria (RAICc and AICc) noticeably outperform the other criteria in general, especially $\mbox{C}_p$ and FPE, particularly for small samples.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/fixedx_VS_hsnr.eps}
  \caption{Results of simulations for variable selection. Fixed-X, high signal. The configurations are the same as in Figure \ref{fig:subsetselection_randomx_hsnr_largep}.}
  \label{fig:subsetselection_fixedx_hsnr_largep}
\end{figure}

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/fixedx_VS_lsnr.eps}
  \caption{Results of simulations for variable selection. Fixed-X, low signal. The configurations are the same as in Figure \ref{fig:subsetselection_randomx_lsnr_largep}.}
  \label{fig:subsetselection_fixedx_lsnr_largep}
\end{figure}

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/fixedx_GR-Ex1.eps}
  \caption{Results of simulations for general restrictions. Fixed-X, GR-Ex1, $\rho=0.5$. The configurations are the same as in Figure \ref{fig:generalrestriction_randomx}.}
  \label{fig:generalrestriction_fixedx}
\end{figure}

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\textwidth]{figures/main/fixedx_GR-Ex4.eps}
  \caption{Results of simulations for general restrictions. Fixed-X, GR-Ex4, $\rho=0.5$. The configurations are the same as in Figure \ref{fig:subsetgeneral_randomx}.}
  \label{fig:subsetgeneral_fixedx}
\end{figure}


%
% CMPT 310: Artificial Intelligence - A Course Overview
% Section: Logical Inference
%
% Author: Jeffrey Leung
%

\section{Logical Inference}
	\label{sec:logical-inference}
\begin{easylist}

& \textbf{Entailment:} Logical connective which means the result is a consequence of the input
	&& Notation: $\alpha \models \beta$

& \textbf{Satisfiability:} Whether a model exists where the logical sentence is true
	&& Includes constraint satisfaction problems
	&& Searches such as: Backtracking, local
	&& \textbf{DPLL algorithm:} Backtracking algorithm which permutes the variables until all clauses are satisfied
		&&& \textbf{Pure symbol:} Symbol which is either never negated or always negated in each clause it appears
		&&& \textbf{Unit clause:} Clause which only contains a single literal
	&& \textbf{WalkSAT algorithm:} Local non-deterministic algorithm which chooses a random false variable and changes it to true to attempt to find a solution
& \textbf{Deduction/Inference:} Given a knowledge base, whether a logical sentence is true (i.e. $KB \models S$)
	&& $KB \models S$ if and only if $KB \land \lnot S$ is unsatisfiable
	&& \textbf{Knowledge base:} Collection of known true sentences

& \textbf{Conjuctive Normal Form (CNF):} Form of a logical statement which is a conjunction of disjunction(s) of literals
	&& Steps to reduce a statement to CNF:
		&&& Replace equivalencies with a conjunction of implications
		&&& Replace implications with $\lnot A \lor B$
		&&& Use DeMorgan's Law to apply negations to clauses
		&&& Use the Distributive Law to reduce clauses to CNF

& \textbf{Inference by resolution:} Process to reduce a set of CNF statements
	&& Given a pair of clauses with complementary literals, resolve them to combine the clauses

\end{easylist}
\clearpage


% Full title as you would like it to appear on the page
\chapter{Error Covariance Fitting and Optimal Telescope Control}
\label{chap:err}
\chaptermark{Error and Control}

\epigraph{By failing to prepare, you are preparing to fail.}{Benjamin Franklin}

In this chapter, we transition from developing our wavefront sensing framework to showing how it can be used to optimally control a telescope like the Rubin Observatory. The connecting bridge is the error covariance matrix of the estimated global wavefront coefficients. In Section \ref{sec:error}, we show how our framework allows this covariance to be easily computed. Then in Section \ref{sec:optcont}, we show how we can use this in control simulations and assess different control strategies.

\section{Error Covariance Fitting}
\label{sec:error}

The key breakthrough is that since the interpolation step of our algorithm has a closed form, the error covariance is a closed form function of the error covariance of our local wavefront predictions from the first step. In order to do this we first fix the positions of the 10 brightest stars on each of the four sensors. For indexing, let $(x_1,y_1)$ be the first position on the first corner sensors, $(x_{11},y_{11})$ be the first position on the second sensor, ... and $(x_{31},y_{31})$ be the first position on the fourth sensor. Then we define $\gamma$ as

\begin{equation*}
\gamma = \begin{bmatrix}
\alpha_4(x_1,y_1)\\
\dots\\
\alpha_4(x_{10}, y_{10})\\
\alpha_5(x_1,y_1)\\
\dots\\
\dots \\
\alpha_{21}(x_{10},y_{10})\\
\alpha_4(x_{11}, y_{11})\\
\dots \\
\dots \\
\alpha_4(x_{21}, y_{21})\\
\dots \\
\dots \\
\alpha_4(x_{31}, y_{31})\\
\dots \\
\dots \\
\alpha_{21}(x_{40},y_{40})\\
\end{bmatrix}
\end{equation*}

\noindent Thus $\gamma \in \mathbb{R}^{720}$, where 720 stems from 10 positions, 4 sensors, and 18 coefficients. We also make the empirically confirmed assumption that the individual local wavefront coefficient predictions are zero centered. Then the error covariance between $\gamma_1, \gamma_2$ measures the error covariance of the $\alpha_4$ predictions across donut locations on the first wavefront sensors; the error covariance between $\gamma_1, \gamma_{11}$ measures the error covariance between the $\alpha_4$ and $\alpha_5$ predictions at the same donut position; the error covariance between $\gamma_1, \gamma_{181}$ measures the error covariance of the $\alpha_4$ predictions between positions on two different corner wavefront sensors. Thus a lot of different forms of covariance are encoded in the covariance between elements of $\gamma$. The full error covariance on the local wavefront predictions is $\Sigma_{\gamma} = \text{Cov}(\gamma, \gamma)$.

We assume that the error $\gamma$ is a multi-dimensional gaussian random variable. Then the covariance from the maximum likelihood estimate is $\Sigma_{\gamma} = (X - E[X])(X-E[X])^T$ where $X \in \mathbb{R}^{497 \times 720}$ such that $X_i = \gamma^{(i)}$ the local wavefront errors from the ith observation. Figure \ref{fig:large-cov} shows the local estimate covariance matrix.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/control/obs_cov_written_thesis.png}
\end{tabular}
\end{center}
\caption[Error Covariance Of Local Wavefront Estimates]{The error covariance matrix $\Sigma_{\gamma}$ for the local wavefront error estimates in units of waves.\label{fig:large-cov}}
\end{figure}

Recall that if $y = Bx$, then $\text{Cov}(y,y) = B\text{Cov}(x,x)B^T$. In our case we can define an $A$, such that $\beta = (A^TA)^{-1}A^T\gamma$. This $A$ is

\begin{equation*}
A = \begin{bmatrix}
I_{18} \otimes \begin{bmatrix}Z_1(x_1,y_1) & Z_2(x_1,y_1) & Z_3(x_1,y_1) \\ \vdots & \vdots & \vdots \\  Z_1(x_{10},y_{10}) & Z_2(x_{10},y_{10}) & Z_3(x_{10},y_{10}) \end{bmatrix}\\
I_{18} \otimes \begin{bmatrix}Z_1(x_{11},y_{11}) & Z_2(x_{11},y_{11}) & Z_3(x_{11},y_{11}) \\ \vdots & \vdots & \vdots \\  Z_1(x_{20},y_{20}) & Z_2(x_{20},y_{20}) & Z_3(x_{20},y_{20}) \end{bmatrix}\\
I_{18} \otimes \begin{bmatrix}Z_1(x_{21},y_{21}) & Z_2(x_{21},y_{21}) & Z_3(x_{21},y_{21}) \\ \vdots & \vdots & \vdots \\  Z_1(x_{30},y_{30}) & Z_2(x_{30},y_{30}) & Z_3(x_{30},y_{30}) \end{bmatrix}\\
I_{18} \otimes \begin{bmatrix}Z_1(x_{31},y_{31}) & Z_2(x_{31},y_{31}) & Z_3(x_{31},y_{31}) \\ \vdots & \vdots & \vdots \\  Z_1(x_{40},y_{40}) & Z_2(x_{40},y_{40}) & Z_3(x_{40},y_{40}) \end{bmatrix}
\end{bmatrix}
\end{equation*}

\noindent where $I_{18} \in \mathbb{R}^{18 \times 18}$ is the identity matrix, $\otimes$ is the kronecker product, and $Z_i(x_j,y_j)$ is the $jth$ circular Zernike polynomial being evaluated at focal plane position $x_j,y_j$. Then the error covariance matrix for $\beta$ is $\Sigma_{\beta} = (A^TA)^{-1}A^T\Sigma_{\gamma}A(A^TA)^{-1}$. It is shown in Figure \ref{fig:small-cov}.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/control/beta_cov_written_thesis.png}
\end{tabular}
\end{center}
\caption[Error Covariance Of Global Wavefront Estimates]{The error covariance matrix $\Sigma_{\beta}$ for the global wavefront error estimates in units of waves.\label{fig:small-cov}}
\end{figure}

The global wavefront covariance $\Sigma_{\beta}$, which can be computed based on the empirically determined $\Sigma_{\gamma}$, is a key advantage to our wavefront sensing framework. We can use this statistical error model to characterize the performance of the system through time.

\section{Optimal Control}
\label{sec:optcont}

We developed a simulator in order to assess different control strategies. The simulator simulates the global wavefront coefficients over discrete time steps. There are three steps at each time step. First, a perturbation is applied. In practice, this would be caused by changes to the wind, gravitational stress, thermal gradients, etc. Second, the global wavefront prediction is made with the error covariance from the previous section. Third a control strategy is applied with this estimate. Figure \ref{fig:control-algo} summarizes the simulator algorithm.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=3in]{figs/control/algo.png}
\end{tabular}
\end{center}
\caption[Control Simulator Algorithm]{The control simulator algorithm. The sequence of perturbations, error covariance matrix, and control strategy are passed by the client.\label{fig:control-algo}}
\end{figure}

We used this simulator to test four different control strategies. The first is the null control strategy which does not make any changes to the control parameters, or

\begin{equation*}
\text{Control-Null}(\hat{\beta}_i) = \hat{\beta}_i
\end{equation*}

\noindent This is useful for null tests.

The second is a gain controller. It makes the update
\begin{equation*}
\text{Control-Gain}(\hat{\beta}_i) = g \cdot \hat{\beta}_i
\end{equation*}
\noindent where $g$ is the gain, which is typically less than 1. When $g=1$, this reduces to the null controller.

The third is a proportional-integral-derivative (PID) controller. It extends the gain controller with two additional additive terms. One term accumulates the estimates, the other uses the difference between the current and previous estimates as a proxy for a derivative. It makes the update

\begin{equation*}
\text{Control-PID}(\hat{\beta}_i) = c_p \cdot \hat{\beta}_i + c_i \sum_{j=1}^i \hat{\beta}_j + c_d \cdot (\hat{\beta}_i - \hat{\beta}_{i+1})
\end{equation*}

\noindent where $c_p, c_i, c_d$ are the configurable model parameters. When $c_i = c_d = 0$ it reduces to the gain controller.

The fourth is a linear-quadratic-gaussian (LQG) controller. In control theory, the separation principle says that the problem of designing an optimal controller for a stochastic system can be decomposed into (i) an optimal estimator of the state and (ii) an optimal deterministic controller. LQG adheres to this decomposition. A Kalman filter (or linear-quadratic state estimator) is used for state estimation and a linear-quadratic regulator is used for control. For our discrete time problem, the update is

\begin{align*}
\text{Control-LQC}(\hat{\beta}_i) &= K_i \cdot x_i\\
x_{i+1} &= x_i - u_i + L_i (\hat{\beta}_i - x_i + u_i)\\
x_0 &= \hat{\beta}_0\\
L_i &= P_i(P_i + \Sigma_{\beta} + V)^{-1}\\
P_{i+1} &= P_{i} - P_i (P_i + \Sigma_{\beta} + V)^{-1} P_i + V\\
P_0 &= \Sigma_{\beta} + V\\
K &= (S_h + R)^{-1} S\\
S_{i+1} &= S_i - S_i (S + R)^{-1}S + Q\\
S_0 &= Q\\
\end{align*}

\noindent where $Q,R,V$ and the horizon $h$ are configurable parameters.

In order to make the sequence of perturbations in the simulator as realistic as possible we developed code to convert contiguous observations from OpSim into sequences of perturbations. We use two types of perturbations: Jump and Box.

Jump perturbations are applied once at a specific time step. They are parameterized by a time step and scale. The 56-dimensional global wavefront perturbation is drawn from a zero-mean multivariate normal distribuation. The covariance is $\Sigma_{\beta}$ times the scale.

Box perturbations are parameterized by a step, duration, and scale. It begins at the step time step and lasts for the duration number of time steps. At each time step, the 56-dimensional global wavefront perturbation is drawn from a zero-mean multivariate normal distribuation. The covariance is $\Sigma_{\beta}$ times the scale. It is worth noting that for each timestep within the duration, a different perturbation is drawn and applied. Table \ref{tab:conversion} shows how we convert OpSim sequences of observations into sequences of perturbations.

\begin{table}
{
\begin{center}
\begin{tabular}{|c|c|}
\hline
Observation & Perturbation\\
\hline
First 50 Observations & Box(start=0, duration=50, scale=2)\\
Last 50 Observations & Box(start=end-50, duration=50, scale=2)\\
Slew $> 5$ & Jump(step=step, scale=5)\\
Slew $> 20$ & Jump(step=step, scale=100)\\
Filter Change & Jump(step=step, scale=20)\\
\hline
\end{tabular}
\end{center}
}
\caption[Observation To Perturbation Conversions]{The mapping between catalog observations and their corresponding perturbations for the control simulator.}
\label{tab:conversion}
\end{table}

Each of the control strategies has parameters. We randomly selected 11 nights within the 10 year observing period for Rubin. We convert each of these nights into a sequence of perturbations for the control simulator. Then we fit the control strategy parameters on the first 10 nights, and evaluate them on the held out final night. The penalty combines the norm of the state, which is zero in the nominal state, and the size of the update.

\begin{equation*}
\text{Penalty} = ||\beta_{i+1}||_2^2 + \frac{1}{2}||\text{Control}(\hat{\beta}_i)||_2^2
\end{equation*}

For the gain controller, we found a gain of 0.39 was optimal. For the PID controller, we found $c_p = 0.37, c_i= 5.4\times 10^{-06}, c_d=.15$ was optimal. For the LQG controller, we found $Q = 0.84 \cdot I_{56}, R=0.88\cdot I_{56}$ with a fixed horizon $h=5$ and $V$ equal to the covariance of the state difference between steps with the null controller on the 10 training nights. The results on the final test night are shown in Figure \ref{fig:control-results}.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/control/control_results.png}
\end{tabular}
\end{center}
\caption[Controller Results]{Left: the penalty of the different controllers through time on the test night. Right: the distributions of the penalties. \label{fig:control-results}}
\end{figure}

The first thing to note is that the performance of the three non-trivial controllers is very similar. The cumulative penalty for the different methods is 1901.78, 16.22, 15.61, and 16.92 for the null, gain, PID, and LQG controllers respectively. It seems like the PID controller best balances the positives from its sophistication and parameterization with the negatives that can arise from overfitting.

As of yet, there has not been an investigation of the tradeoffs between different control strategis for Rubin. The current plan is to spend time during commissioning to take data and develop a strategy on the mountain. However, each extra day on the mountain not only delays the survey but also runs up the project budget. The framework we have presented here
has the potential to save this critical commissioning time by enabling this work on the mountain to be done now, with this framework.


We conduct experiments using three neural network types and three types of NLP tasks, described in \S\ref{sec:1}, \S\ref{sec:2}, and \S\ref{sec:3} below.
\subsection{MLP \& Sentence Classification}\label{sec:1}
\paragraph{Model}
We experiment with a multi-layer perceptron (MLP) applied to sentence-level classification tasks. That is, input to the MLP is a sentence or short text, represented as a fixed-size vector embedding. The output of the MLP is a label which classifies the sentence or short text. We use two sentence representation techniques, namely, Sent2Vec \cite{Pagliardini:2018}, of dimensionality 600, and InferSent \cite{Conneau:2017}, of dimensionality 4096. Our MLP has the form:
\begin{align*}
  \mathbf{x}_i &= f(\mathbf{x}_{i-1}\cdot \mathbf{W}_i+\mathbf{b}_i)\\
  \mathbf{y} &= \text{softmax}(\mathbf{x}_{N}\mathbf{W}_{N+1}+\mathbf{b}_{N+1})
\end{align*}
where $\mathbf{x}_0$ is the input representation, $\mathbf{x}_1,\ldots,\mathbf{x}_{N}$ are hidden layer representations, and $\mathbf{y}$ is the output, a probability distribution over the classes in the classification task. Vectors $\mathbf{b}$ and matrices $\mathbf{W}$ are the learnable parameters of our network. The activation function is given by $f$ and ranges over the choices described in \S\ref{sec:theory}.
\paragraph{Data}
We use four sentence classification tasks, namely: movie review classification (MR), subjectivitiy classification (SUBJ),
question type classification (TREC),
and classifying whether a sentence contains an argumentation structure of a certain type (claim, premise, major claim) or else is non-argumentative (AM). The first three datasets are standard sentence classification datasets and contained in the SentEval framework.\footnote{\url{https://github.com/facebookresearch/SentEval}} We choose the AM dataset for task diversity, and derive %the AM dataset
it by projecting token-level annotations in the dataset from \citet{Stab:2017} to the sentence level. In the rare case ($<$5\% of the cases) when a sentence contains multiple argument types, we choose one based on the ordering Major Claim (MC) $>$ Claim (C) $>$ Premise (P). Datasets and examples are listed in Table \ref{table:data_sent}.

\input{tables/tasks}

\paragraph{Approach}
We consider 7 ``mini-experiments'':
\begin{itemize}[noitemsep,leftmargin=0.6cm]
  \item (1): MR dataset with Sent2Vec-unigram embeddings as input and 1\% of the full data as training data; (2): the same mini-experiment with 50\% of the full data as training data. In both cases, the dev set comprises 10\% of the full data and the rest is for testing.
  \item (3,4): SUBJ with InferSent embeddings and likewise 1\% and 50\% of the full data as train data, respectively.
  \item (5): The TREC dataset with original split in train and test; 50\% of the train split is used as dev data.
  \item (6): The AM dataset with original split in train, dev, and test \cite{Eger:2017}, and with InferSent input embeddings. (7): the same mini-experiment with Sent2Vec-unigram embeddings.
\end{itemize}
We report accuracy for mini-experiments (1-5) and macro-F1 for (6-7). We report macro-F1 for (6-7) because the AM dataset is imbalanced.

%Some setups, such as the input representation, were randomly matched to the tasks.
The motivation behind choosing different input embeddings for different tasks was to investigate a wider variety of conditions.
%The motivation behind using subsets of the full data for training in case of MR and SUBJ was to induce diverse data size conditions;
Choosing subsets of the full data had the same intention.
%to see if the choice of activation function matters (differently) under different data size conditions;
%we did this for MR and SUBJ because, unlike AM and TREC, they have no pre-specified splits into train and test.
%The overall goal of varying some of these parameters (train/test splits, input embeddings) was to explore a larger search space.

For all 7 mini-experiments, we draw the same 200 randomly chosen hyperparameters from the ranges indicated in Table \ref{table:hyperparams_sent}.
%The hyperparameters concern the optimizer chosen for the MLP; the number $N$ of hidden layers; the dropout value \cite{Srivastava:2014} applied on each hidden layer; the number of hidden units in each hidden layer (identical numbers across all layers); the learning rate; and the initializer.
All experiments are conducted in keras.\footnote{\url{https://keras.io/}} %Each of the 200 hyperparameter configurations is run with 5 random initializations of weight matrices and the average is stored.

For each of the 21 different activation functions detailed in \S\ref{sec:theory}, we conduct each mini-experiment with the 200 randomly chosen hyperparameters. All activation functions use the same hyperparameters and the same train, dev, and test splits.

%Overall, we run $21\times 200\times 7\times 5 = 147,000$ experiments.

We store two results for each mini-experiment, namely: (i) the test result corresponding to the \textbf{best} (\best) dev performance; (ii) the \textbf{average} (\avg) test result across all hyperparameters. The \best{} result scenario mirrors standard optimization in machine learning: it indicates the score one can obtain with an activation function when %it is used in
the MLP is well-optimized. The \avg{} result scenario is an indicator for what one can expect when hyperparameter optimization is `shallow' (e.g., because computing times are prohibitive): it gives the average performance for randomly chosen hyperparameters. We note that we run each hyperparameter combination %is run
with 5 different random weight initializations and all the reported scores (best dev score, best \best{}, best \avg{}) are averages over these 5 random initializations.

Finally, we set the following hyperparameters for all MLP experiments: patience of 10 for early stopping, batch size 16, 100 epochs for training.

\input{tables/hyperparams_sent}

\paragraph{Results}
Figure \ref{fig:sent} shows \best{} and \avg{} results, averaged over all 7 mini-experiments, for each activation function. To make individual scores comparable across mini-experiments, we perform max normalization and divide each score by the maximum score achieved in any given mini-experiment (for \best{} and \avg{}, respectively) before averaging.\footnote{We chose max normalization so that certain tasks/mini-experiments would not unduly dominate our averaged scores. Overall, our averaged scores are not (much) affected by this decision, however: the Spearman correlation of rankings of activation functions under max normalization and under no max normalization are above 0.98 in all our three classification scenarios considered in \S\ref{sec:1},\S\ref{sec:2},\S\ref{sec:3}.}

For \best{}, the top performers are the rectifier functions (\relu{}, \lrelua{}, \prelu{}) as well as maxout and \pentan{}. The newly discovered activation functions lag behind, with the best of them being \minsin{} and \swish{}. \linear{} is worst, together with \elu{} and \cube. Overall, %we see, however, that
the difference between the best activation function, \relu{}, and the worst, \linear, is only roughly 2pp, however. This means that if hyperparameter search is done carefully,  the choice of activation function is less important for these sentence classification tasks. Particularly the (binary) tasks MR and SUBJ appear robust against the choice of activation function, with the difference between the best and worst function being always less than 1pp, in all settings. For TREC and AM, the situation is slightly different: for TREC, the difference is 2pp (\swish{} vs.\ \maxsig) and for AM, it is 3pp using InferSent embeddings (\swish{} vs.\ \cube) and 12pp using Sent2Vec embeddings (\relu{} vs.\  \linear). It is noteworthy that \swish{} wins 2 out of 3 cases in which the choice of activation function really matters.
\begin{figure}[htb]
  \centering
  %\scalebox{0.5}{\input{plots/sent3.tex}}
  \scalebox{0.5}{\input{plots/sent.tex}}
  \caption{Sentence Classification. Left y-axis: \best. Right y-axis: \avg{}. Score on y-axes is the average over all mini-experiments.}
  \label{fig:sent}
\end{figure}

\avg{} results are very different from \best{} results. Here, somewhat surprisingly, the oscillating \mysin{} function wins, followed by \pentan{}, maxout and \swish. The difference between the best \avg{} function, \mysin, and the worst, \cube, is more than 30pp. This means that using \cube{} is much riskier and requires more careful hyperparameter search compared to \mysin{} and the other top performers.
%Figure \ref{fig:sent} also shows that better \best{} performers tend to be better \avg{} performers, and vice versa.
%have better average performance and vice versa.

\subsection{CNN \& Document Classification}\label{sec:2}
\paragraph{Model} Our second paradigm is document classification using a CNN. This approach has been popularized in NLP by the ground-breaking work of \citet{Kim:2014}. Even though shallow CNNs do not reach state-of-the-art results on large datasets anymore \cite{Johnson:2017}, simple approaches like (shallow) CNNs are still very competitive for smaller datasets \cite{Joulin:2016}.

Our model operates on token-level and first embeds a sequence of tokens $x_1,\ldots,x_n$, represented as 1-hot vectors, into learnable embeddings $\mathbf{x}_1,\ldots,\mathbf{x}_n$. The model then applies 1D-convolution on top of these embeddings. That is, a filter $\mathbf{w}$ of size $h$ takes $h$ successive embeddings $\mathbf{x}_{i:i+h-1}$, performs a scalar product and obtains a feature $c_i$:
\begin{align*}
  c_i = f(\mathbf{w}\cdot \mathbf{x}_{i:i+h-1}+b).
\end{align*}
Here, $f$ is the activation function and $b$ is a bias term. We take the number $n_k$ of different filters as a hyperparameter. When our network has multiple layers, we stack another convolutional layer on top of the first (in total we have $n_k$ outputs at each time step), and so on. Our penultimate layer is a global max pooling layers that selects the maximum from each feature map. A final softmax layer terminates the network.
\paragraph{Data} We use two document classification tasks, namely: 20 Newsgroup (NG) and Reuters-21578 R8 (R8). Both datasets are standard document classification datasets. In NG, the goal is to classify each document into one of 20 newsgroup classes (alt.atheism, sci.med, sci.space, etc.). In R8, the goal is to classify Reuters news text into one of eight classes (crude, earn, grain, interest, etc.).  We used the preprocessed files from \url{https://www.cs.umb.edu/~smimarog/textmining/datasets/} (in particular, stopwords are removed and the text is stemmed).
\paragraph{Approach}
We consider 4 mini-experiments:
\begin{itemize}[noitemsep,leftmargin=0.6cm]
\item (1,2) NG dataset with 5\% and 50\%, respectively of the full data as train data. In both cases, 10\% of the full data is used as dev data, and the rest as test data.
\item (3,4) Same as (1,2) for R8.
%R8 dataset with same specifications as for NG.
\end{itemize}
We report accuracy for all experiments. We use a batch size of 64, 50 epochs for training, and a patience of 10. For all mini-experiments, we again draw 200 randomly chosen hyperparameters from the ranges indicated in Table \ref{table:hyperparams_sent}.
%As for the train/dev/test splits,
The hyperparameters and train/dev/test splits are the same for all activation functions.

\paragraph{Results}
Figure \ref{fig:doc} shows \best{} and \avg{} results, averaged over all mini-experiments. This time, the winners for \best{} are \elu{}, \selu{} (again two members from the rectifier family), and \maxoutb{}, but the difference between \maxoutb{} and several lower ranked functions is minimal. The \cube{} function is again worst and \sigmoid{} and \cosid{} have similarly bad performance. Except for \minsin{}, the newly proposed activation functions from \citet{Ramach:2018} again considerably lag behind. The most stable activation functions are the maxout functions as well as \pentan{}, \mytanh{} and \mysin{}.

\begin{figure}[!htb]
\centering
%\scalebox{0.5}{\input{plots/doc2}}
\scalebox{0.5}{\input{plots/doc}}
\caption{Doc classification.}
\label{fig:doc}
\end{figure}

\subsection{RNN \& Sequence Tagging}\label{sec:3}
\paragraph{Model} Our third paradigm is sequence tagging, a ubiquitous model type in NLP. In sequence tagging, a sequence of input tokens $w_1,\ldots,w_K$ is mapped to a sequence of labels $y_1,\ldots,y_K$. Classical sequence tagging tasks include POS tagging, chunking, NER, discourse parsing \cite{Braud:2017}, and argumentation mining \cite{Eger:2017,Schulz:2018}. We use a standard recurrent net for sequence tagging, whose form is:
\begin{align*}
  \mathbf{h}_i &= f(\mathbf{h}_{i-1}\mathbf{W}+\mathbf{w}_i\cdot\mathbf{U}+\mathbf{b})\\
  \mathbf{y}_i &= \text{softmax}(\mathbf{h}_i\mathbf{V}+\mathbf{c})
\end{align*}
Here, $\mathbf{w}_i$ are (pre-trained) word embeddings of words $w_i$. Vectors $\mathbf{b},\mathbf{c}$ and matrices $\mathbf{U},\mathbf{V},\mathbf{W}$ are parameters to be learned during training. The above describes an RNN with only one hidden layer, $\mathbf{h}_i$, at each time step, but we consider the generalized form with $N\ge 1$ hidden layers; we also choose a bidirectional RNN in which the hidden outputs of a forward RNN and a backward RNN are combined.
RNNs are particularly deep networks---indeed, the depth of the network corresponds to the length of the input sequence---which makes them particularly susceptible to the vanishing gradient problem \cite{Pascanu:2013}.

%[Say a word on LSTMs]
Initially, we do not consider the more popular LSTMs here for reasons indicated below. However, we include a comparison after discussing the RNN performance.
%for two reasons: (i) activation functions in LSTMs usually take other roles than in standard RNNs, namely, that of gating (gate is open, closed, or in-between), and thus, the expectation is that the choice of activation function is either not as important or restricted to certain `binary-like' function; (ii) with proper initialization, standard RNNs have been shown to be able to perform as well, or even better, than LSTMs \cite{Le:2015}. However, we will also compare RNNs to LSTMs, albeit in fewer scenarios.

\paragraph{Data}
We use two sequence tagging tasks, namely: English POS tagging (POS), and token-level argumentation mining (TL-AM) using the same dataset (consisting of student essays) as for the sentence level experiments. In token-level AM, we tag each token with a BIO-label plus the component type, i.e., the label space is $\mathcal{Y}=\{\text{B},\text{I}\}\times\{\text{MC},\text{C},\text{P}\}\cup\{\text{O}\}$, where
%MC stands for major claim, C for claim, and P for premise, and `
`O'' is a label for non-argumentative tokens. The motivation for using TL-AM is that, putatively, AM has more long-range dependencies than POS or similar sequence tagging tasks such as NER, because argument components are much longer than named entities and component labels also %do not
depend less %very much
on the current token.

%Dataset statistics are shown in Table \ref{table:data_sent}.
\paragraph{Approach} We consider 6 mini-experiments:
\begin{itemize}[noitemsep,leftmargin=0.6cm]
  \item (1): TL-AM with Glove-100d word embeddings and 5\% of the original training data as train data; (2) the same with 30\% of the original training data as train data. In both cases, dev and test follow the original train splits \cite{Eger:2017}.
  \item (3,4) Same as (1) and (2) but with 300d Levy word embeddings \cite{Levy:2014}.
  \item (5,6): POS with Glove-100d word embeddings and 5\% and 30\%, respectively, of the train data of a pre-determined train/dev/test split (13k/13k/178k tokens). Dev and test are fixed in both cases.
\end{itemize}
We report macro-F1 for mini-experiments (1-4) and accuracy for (5-6). For our RNN implementations, we use the accompanying code of (the state-of-the-art model of) \citet{Reimers:2017}, which is implemented in keras. The network uses a CRF layer as an output layer. We use a batch size of 32, train for 50 epochs and use a patience of 5 for early stopping.
\paragraph{Results}
Figure \ref{fig:seq} shows \best{} and \avg{} results, averaged over all 6 mini-experiments, for each activation function. We exclude \prelu{} and the maxout functions because the keras implementation does not natively support these activation functions for RNNs. We also exclude the \cube{} function because it performed very badly.  \begin{figure}[!htb]
\centering
\scalebox{0.5}{
%\input{plots/seq2}}
\input{plots/seq}}
\caption{Sequence tagging.}
\label{fig:seq}
\end{figure}

Unlike for sentence classification, there are much larger differences between the activation functions. For example, there is almost 20pp difference between the best \best{} activation functions: \relu{}, \lrelua{}, \swish{}, \pentan{}, and the worst ones: \linear{}, \cosid{}, and \sigmoid{} (the differences were larger had we included \cube{}). Interestingly, this difference is mostly due to the TL-AM task: for POS, there is only 3pp difference between the best function (\sigmoid{} (sic!), though with almost zero margin to the next best ones) and the worst one (\linear), while this difference is almost 40pp for TL-AM.
%If we follow conventional wisdom, then we would speculate that POS tagging is a too simple task where the label for a token only depends on the current token and some small context. Thus, the task apparently lacks long-range dependencies, as we had reasoned, which may cause different activation functions to have little influence on performance.
This appears to confirm our concerns regarding the POS tagging task as not being challenging enough due to lack of, e.g., long-range dependencies.

The four best \best{} activation functions in Figure \ref{fig:seq} are also the functions with the best \avg{} results, i.e., they are most stable over different hyperparameters. %choices.
The clear winner in this category is \pentan{} with 100\% \avg{} score, followed by \swish{} with 91\%. Worst is \cosid{} with 30\%. It is remarkable how large the difference between \mytanh{} and \pentan{} is both for \best{} and \avg{}---7pp and 20pp, respectively, which is much larger than the differences between the analogous pair of LReLU and \relu{}. This appears to make a strong case for the importance of the slope around the origin, as suggested in \citet{Xu:2016}.
%, lacking, e.g., long-range dependencies

%\begin{table}[!htb]
%  \centering
%  \begin{tabular}{lrr}
%    \toprule
%    Function & Best & Avg \\ \midrule
%    \relu & 52.95 \\
%    \lrelua & 52.92 \\
%    \swish & 52.66 \\
%    \lrelub & 52.19 \\
%    \pentan & 51.76 \\
%    \arctid & 50.23 \\
%    \midrule
%    \elu & 48.72 \\
%    \minsin & 48.43 \\
%    \maxtanh & 47.42 \\
%    \mytanh & 46.74 \\
%    \mysin & 46.00 \\
%    \maxsig & 45.54 & \\
%    \selu & 44.72 & \\
%    \linear & 42.20 & \\
%    \midrule
%    \sigmoid & 39.58 & \\
%    \cosid & 34.76 & \\
%    \cube & 11.33 & \\
%  \end{tabular}
%  \caption{ArgMin PE, macro-F1 scores %in \%.}
%  \label{table:seq_pe}
%\end{table}

\paragraph{LSTM vs.\ RNN} Besides an RNN, we also implemented a more popular RNN model with (bidirectional) LSTM blocks in place of standard hidden layers. Standard LSTM units follow the equations (simplified):
\begin{align*}
  \mathbf{f}_t &= \sigma([\mathbf{h}_{t-1};\mathbf{x}_t]\cdot \mathbf{W}_f),\\
  \mathbf{i}_t &= \sigma([\mathbf{h}_{t-1};\mathbf{x}_t]\cdot \mathbf{W}_i),\\
  %&\vdots \\
  \mathbf{o}_t &= \sigma([\mathbf{h}_{t-1};\mathbf{x}_t]\cdot \mathbf{W}_o)\\
  \mathbf{c}_t &= \mathbf{f}_t\odot \mathbf{c}_{t-1}+\mathbf{i}_t\odot\tau([\mathbf{h}_{t-1};\mathbf{x}_t]\cdot \mathbf{W}_c)\\
  \mathbf{h}_t &= \mathbf{o}_t\odot \tau(\mathbf{c}_t),
\end{align*}
where $\mathbf{f}_t$ and $\mathbf{i}_t$ are perceived of as \emph{gates} that control information flow,
%(further gates and equations skipped for readability),
$\mathbf{x}_t$ is the input at time $t$ and $\mathbf{h}_t$ is the hidden layer activation. In keras (and most standard references), $\sigma$ is the (hard) sigmoid function, and $\tau$ is the \mytanh{} function.

We ran an LSTM on the TL-AM dataset with Levy word embeddings and 5\% and 30\% data size setup. We varied $\sigma$ and $\tau$ independently, keeping the respective other function at its default.

We find that the top two choices for $\tau$ are \pentan{} and \mytanh{} (margin of 10pp), given that $\sigma$ is sigmoid. For $\tau=$ \mytanh{}, the best choices are $\sigma=$ \pentan{}, \sigmoid, and \mytanh{}. All other functions perform considerably worse. Thus, the top-performers are all saturating functions, indicating the different roles activation functions play in LSTMs---those of gates---compared to standard layers. It is worth mentioning that choosing $\sigma$ or $\tau$ as \pentan{} is on average better than the standard choices for $\sigma$ and $\tau$. %[Moreover, when we choose both $\sigma$ and $\tau$ as \pentan{}, then ...]
Indeed, choosing $\tau=\sigma=$ \pentan{} is on average 2pp better than the default choices of $\tau,\sigma$.
%and is on par with the choice $\tau=$ \sigmoid{} and $\sigma=$ \pentan{}.

It is further worth mentioning that the best \best{} results for the LSTM are roughly 5pp better (absolute) than the best corresponding choices for the simple RNN.


\section{Discussion and conclusion}

From our empirical results we are able to verify the stability properties of the Hopfield network. The network is able to restore multiple different patterns from partial inputs, and is stable in the equlibrium points. Our empirical results cannot guarantee global stability, and it is difficult to draw good conclusions purely from a few simulated results. There are however a great deal of research on the stability of the Hopfield network, and the stability of the network have been mathematically proved using Lyapunov analysis. By considering the energy in \cref{eq:energy} as a Lyapunov candidate, it can be show that the energy is (nonstrictly) decreasing when using the update rule in \cref{eq:update} \cite{lyapnuv-stability}.

When simulating with multiple uncorrelated patterns the network energy lower limit from \cref{eq:energy-limits} appears to be roughly divided equally between the two patterns. The energy limits may therefore be used to quantify the capacity (i.e. the number of memories the network can store). How the energy is distributed does however depend on how similar the memories are. We can think of memories with a lot in common as creating smaller valleys within a larger valley, and thereby sharing some potential energy. Intuitively we should in theory be able to store as many memories as we can create distinct valleys using the finite amount of discrete energy in the network - assuming the network is initialized perfectly without noise.

The energy may also be used to quantify the robustness of the network. By considering the amount of work needed (in a physical sense) to move the state of the network from one memory to another, we can create a measure for how robust the network is to random perturbations. If the amount of energy required to move the state between two memories (i.e. the amount of work required to move the state out of one valley) is small, then noise in only a few bits may be enough to recall the wrong memory. Storing too many memories (i.e. having too many small valleys), or storing very similar patterns will therefore decrease the robustness, as was seen in our simulations in \cref{fig:multiple-capacity}.

Our results also showed that for each stored pattern, there are two equlibrium points - an original and an inverted pattern. In other words, the Hopfield network does not store the actual values of the stored patterns, but only whether any pair of neurons should be equal or not. This appears to be consistent with how we as humans are able to associate memories with objects regardless of specific details - we can recognize a written number regardless of color and background.

There are however many limitations to the Hopfield network, and it is in no way a perfect model of the mechanisms behind memory in humans or animals. Nevertheless it helps us understand how a complex network of simple units (neurons) are able to recall memories, and many of its limitations have been solved in more modern approaches to ANNs.