text:
\section{Introduction and main results}

Sets of points on the sphere $\mathbb{S}^{2}$ that are well-distributed in some sense conform an interesting object of study, see for example \cite{Brauchart2015293} for an interesting survey with different approaches to well-distributed points. One usually seeks for points with small {\em cap discrepancy} or maximal {\em separation distance} or, as we do in this paper, minimal {\em potential energy}.

\subsection{Riesz and logarithmic energy}
Given $s\in(0,\infty)$, the Riesz potential or $s$--energy of a set on points $\omega_{N} = \{ x_{1}, \ldots  ,x_{N} \}$ on the sphere $\mathbb{S}^{2}$ is
\begin{equation}\label{eq:Riesz}
\E{s}(\omega_{N}) = \displaystyle\sum_{i \neq j} \frac{1}{\| x_{i} - x_{j} \|^{s}}.
\end{equation}
This energy has a physical interpretation for some particular values of $s$, i.e. for $s=1$ the Riesz energy is the Coulomb potential and in the special case $s=0$ the energy is defined by
\[
\E{\log}(\omega_{N})=\left. \frac{d}{ds} \right|_{s=0}\E{s}(\omega_N) = \sum_{i \neq j} \log\| x_{i} - x_{j} \|^{-1}
\]
and is related to the transfinite diameter and the capacity of the set by classical potential theory, see for example \cite{doohovskoy2011foundations}.
\subsection{Smale's 7th problem}
Shub and Smale \cite{SS93} found a relation between the condition number (a quantity measuring the sensitivity to zero finding) of polynomials and the logarithmic energy of associated spherical points. Inspired by this relation, they proposed a problem that is nowadays known as Smale's 7th problem \cite{Sm2000}: find a constructive (and fast) way to produce $N$ points with quasioptimal logarithmic energy. More exactly, on input $N$, one must produce a set of $N$ points $\omega_N$ on the unit sphere such that
\[
\E{\log}(\omega_{N})-m_N\leq c\log N,
\]
where $c$ is some universal constant and $m_N$ is the minimum possible value of $\E{\log}$ among all collections of $N$ spherical points.

\subsection{The value of $m_N$} A major difficulty in Smale's 7th problem is that the value of $m_N$ is not even known up to precision $\log N$. A series of papers \cite{Wagner,RSZ94,Dubickas,Brauchart2008} gave upper and lower bounds for the value of $m_N$. The last word has been given in \cite{BS18} where this value is related to the minimum renormalized energy introduced in \cite{SS15} proving the existence of an $O(N)$ term. The current knowledge is:
\begin{equation}\label{eq:as}
m_N=W_{\log}(\S2)\,N^2-\frac12\,N\log N+C_{\log}\,N+o(N),
\end{equation}
where
\begin{equation}\label{conjetura}
W_{\log}(\S2)=\frac{1}{(4\pi)^2}\int_{x,y\in\S2}\log\|x-y\|^{-1}\,d(x,y)=\frac{1}{2}-\log 2
\end{equation}
is the continuous energy and $C_{\log}$ is a constant. Combining  \cite{Dubickas} with \cite{BS18} it is known that
\[
-0.2232823526\ldots\leq C_{\log}\leq 2\log 2 +\frac12\log\frac23+3\log\frac{\sqrt\pi}{\Gamma(1/3)}=-0.0556053\ldots,
\]
and indeed the upper bound for $C_{\log}$ has been conjectured to be an equality using two different approaches \cite{BHS12,BS18}.
\subsection{Explicit constructions towards Smale's 7th problem}
Several point sequences that seem to have low logarithmic energy have been proposed. In \cite{dolomites} we find a number of families of points (some of them are random and some of them are not) together with numerical evidence of their properties for values as high as $N=50.000$ spherical points.  However, obtaining theoretical results for the properties of these sequences has proved a very hard task. Of course one can just run a generic optimization algorithm starting on those sequences and get seemingly optimal collections of points but theoretical results about the asymptotical properties of the output of such methods are quite out of reach.

Theoretical computations of the energy of constructively feasible families of points has only been done in a few cases.

Points coming from the spherical ensemble (that can be seen after a stereographic projection as the eigenvalues of $A^{-1}B$ where $A$ and $B$ are random Gaussian matrices, see \cite{krishnapur2009}) have been proved in \cite{EJP3733} to have average logarithmic energy
\[
W_{\log}(\S2)\,N^2-\frac12\,N\log N+c_1\,N+o(N),
\]
where $c_1=\log 2-\gamma/2=0.404539348109\ldots$ (here, $\gamma$ is the Euler--Mascheroni constant).

On the other hand, points obtained (after the stereographic projection) as zeros of certain random polynomials have been studied in \cite{ABS11} proving that the expected value of the logarithmic energy in this case is
\[
W_{\log}(\S2)\,N^2-\frac12\,N\log N+c_2\,N+o(N),
\]
where $c_2=-W_{\log}(\S2)=0.1931471805\ldots$

Both $c_1$ and $c_2$ are quite far from the known upper bound for $C_{\log}$ and thus also far from providing an answer to Smale's problem.

\subsection{Main result: the Diamond ensemble}

In this paper, we define a collection of random points, the {\em Diamond ensemble} $\diamond(N)$, depending on several parameters. For appropriate choices of the parameters, our construction produces families of points that very much resemble some already known families for which the asymptotic expansion of the logarithmic energy is unknown, such as the octahedral points or the zonal equal area nodes, see \cites{RSZ94, HOLHOS20141092,  dolomites}.
Indeed our paper can be seen as a follow up of \cite[Theorem 3.2]{RSZ94}.

A quasioptimal choice of these parameters is described in Section \ref{sec:qode}, we call the resulting set the {\em quasioptimal Diamond ensemble}, and its main interest is that we can prove the following bound.

\begin{thm}\label{ThmPpal2}

The expected value of the logarithmic energy of the quasioptimal Diamond ensemble described in Section \ref{sec:qode} is
	\begin{equation*}
	W_{\log}(\S2)\,N^2-\frac12\,N\log N+c_\diamond\,N+o(N),
	\end{equation*}
	where $c_\diamond=- 0.0492220914515784\ldots$ satisfies
	\begin{multline*}
	14340\,c_\diamond=19120\log239 - 2270\log227 - 1460\log73 - 265\log53
	- 1935\log43\\ - 930\log31 - 1710\log19 - 1938\log17 + 19825\log13
	+ 1750\log7 \\- 4250\log5 - 131307\log3 + 56586\log2 - 7170.
	\end{multline*}
\end{thm}
The value of the constant is thus approximately $0.0058$ far from the valued conjectured in \eqref{conjetura}. The Diamond ensemble is fully constructive: once a set of parameters is chosen, one just has to choose some uniform random numbers $\theta_1,\ldots,\theta_p\in[0,2\pi]$ and then the $N$ points are simply given by the direct formulas shown in Section \ref{sec:qode}. It is thus extremely easy to generate these sequences of points.

As one can guess from the expression of $c_\diamond$, obtaining the exact value for that constant requires the computation of a huge number of elementary integrals and derivatives and has been done using the computer algebra package {\tt Maxima}.
Our proof of Theorem \ref{ThmPpal2} is thus, in some sense, a computer aided proof.
A more simple example (with more simple parameters) that can actually be done by hand is presented in Section \ref{sec:simple}.

\subsection{Structure of the paper}

In Section \ref{formula} we present a formula for computing the energy of the roots of unity of some parallels.
In Section \ref{diamond} we define the Diamond ensemble and through the formula of Section \ref{formula} we compute its associated logarithmic energy.
In Section \ref{examples} we present some concrete examples of the Diamond ensemble.
In particular a simple model that can be made by hand, a more elaborated example and the quasioptimal Diamond ensemble in terms of minimizing logarithmic energy.
In this section we also give the asymptotic expansion of the logarithmic energy associated to every single example.
Section \ref{section_proofs} is devoted to proofs and
Appendix \ref{appendix} contains some bounds for the error of the trapezoidal rule.

\section{A general construction and a formula for its average logarithmic energy}\label{formula}
Fix $z\in(-1,1)$. The parallel of height $z$ in the sphere $\mathbb{S}^{2} \subset \mathbb{R}^{3}$ is simply the set of points $x\in\S2$ such that $\langle x,(0,0,1)\rangle=z$. A general construction of points can then be done as follows:
\begin{enumerate}
	\item Choose a positive integer $p$ and $z_1,\ldots,z_p\in(-1,1)$. Consider the $p$ parallels with heights $z_1,\ldots,z_p$.
	\item For each $j$, $1\leq j\leq p$, choose a number $r_j$ of points to be allocated on parallel $j$.
	\item Allocate $r_j$ points in parallel $j$ (which is a circumference) by projecting the $r_j$ roots of unity onto the circumference and rotating them by random phase $\theta_j\in[0,2\pi]$.
	\item To the already constructed collection of points, add the North and South pole $(0,0,1)$ and $(0,0,-1)$.
\end{enumerate}
We will denote this random set by $\Omega(p,r_{j},z_{j})$. Explicit formulas for this construction are easily produced: points in parallel of height $z_j$ are of the form
\begin{equation}\label{points}
\begin{split}
& x = \left( \sqrt{1 - z_{j}^{2}} \cos \theta, \sqrt{1 - z_{j}^{2}} \sin \theta, z_{j} \right) \\
\end{split}
\end{equation}
for some $\theta\in[0,2\pi]$ and thus the set we have described agrees with the following definition.
\begin{defn}\label{DefnOmegaGeneral}

	Let $\Omega(p,r_{j},z_{j})$ be the following set of points

	\begin{equation}\label{DefOmega}
	\Omega(p,r_{j},z_j) =
	\begin{cases}
	\mathcal{N} = (0,0,1) \\
	x_{j}^{i} = \left( \sqrt{1-z_{j}^{2}}\cos\left( \frac{2\pi i}{r_{j}} + \theta_{j} \right), \sqrt{1-z_{j}^{2}}\sin\left( \frac{2\pi i}{r_{j}}  + \theta_{j}\right), z_{j} \right) \\
	\mathcal{S} = (0,0,-1)
	\end{cases}
	\end{equation}

	\noindent where $r_{j}$ is the number of roots of unity that we consider in the parallel $j$, $1 \leq j \leq p$ is the number of parallels, $1 \leq i \leq r_{j}$ and $0 \leq \theta_{j} < 2\pi$ is a random angle rotation in the parallel $j$.

\end{defn}

The following proposition is easy to prove.

\begin{prop}\label{PropAveEner}
Let $x$ be chosen uniformly and randomly in the parallel of height $z_i$ and let $y$ be chosen uniformly and randomly in the parallel of height $z_j$. The average of the logarithmic energy associated to $x$ and $y$ is
\begin{equation*}
\begin{split}

- \frac{\log \left( 1 - z_{i}z_{j} + |z_{i} - z_{j}| \right)}{2}.
\end{split}
\end{equation*}
\end{prop}
The following result follows directly from Proposition \ref{PropAveEner}.

\begin{cor}\label{CorAveEner2}
Let $x_j^i$ be as in Definition \ref{DefnOmegaGeneral}. Then,
\begin{equation*}
\begin{split}
 \Esp{\theta_{j},\theta_{k}}{-\sum_{l=1}^{r_{k}} \sum_{i=1}^{r_{j}} \log \left( || x_{j}^{i} - x_{k}^{l}|| \right)}
 =
- r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2},
\end{split}
\end{equation*}
where $\theta_j,\theta_k$ are uniformly distributed in $[0,2\pi]$.
\end{cor}

From Corollary \ref{CorAveEner2} we will prove the following result which gives us an expression for the expected logarithmic energy of the set $\Omega(p,r_{j},z_j)$.
\begin{prop}\label{PropEnergyOmegaGeneral}
The average logarithmic energy of points drawn from $\Omega(p,r_{j},z_j)$ is

\begin{multline*}
\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j},z_j))}
 =\\-2\log(2)
-\sum_{j=1}^{p}
r_{j}\left[
\log(4)
+ \frac{1}{2}  \log(1 - z_{j}^{2})
+ \log r_{j}\right]\\
- \sum_{j,k=1}^{p}  r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}.
\end{multline*}

\end{prop}
It turns out that, for any fixed choice of $r_1,\ldots,r_p$, one can compute exactly the optimal choice of the heights $z_1,\ldots,z_p$.
\begin{prop}\label{Propminparallel}
Given $\{ r_{1},...,r_{p} \}$ such that $r_{i} \in \mathbb{N}$, there exists a unique set of heights $\{z_{1},\ldots,z_{p} \}$ such that $z_{1} > \ldots > z_{p}$ and $\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j},z_j))}
$ is minimized.
The heights are:
\begin{equation*}
z_{l}
=
\frac{\displaystyle\sum_{j=l+1}^{p}r_{j} - \displaystyle\sum_{j=1}^{l-1}r_{j}}{1 + \displaystyle\sum_{j=1}^{p}r_{j}}=1-\frac{1+r_l+2\sum_{j=1}^{l-1}r_j}{N-1},
\end{equation*}
where $N=2+\sum_{j=1}^{p}r_j$ is the total number of points.
\end{prop}

From now on we will denote by $\Omega(p,r_{j})$ the set $\Omega(p,r_{j},z_{j})$ where the $z_{j}$ are chosen as in Proposition \ref{Propminparallel}.
With this choice of $z_j$ we have the main result of this section:
\begin{thm}\label{cor:nuevasuma}
Let $p=2M-1$ be an odd integer.
If $r_j=r_{p+1-j}$ and $z_j$ are chosen as in Proposition \ref{Propminparallel} we then have
	\begin{equation*}
	\begin{split}
	&
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j}))}
	\\
	& =
	-(N-1)\log(4)-\sum_{j=1}^{p}r_{j}\log r_{j}-(N-1)\sum_{j=1}^pr_j(1-z_j)\log(1-z_j)
	\\
	& =
	-(N-1)\log(4)+r_M\log r_M-2\sum_{j=1}^{M}r_{j}\log r_{j}\\&-(N-1)\sum_{j=1}^Mr_j(1-z_j)\log(1-z_j)-(N-1)\sum_{j=1}^Mr_j(1+z_j)\log(1+z_j).
	\end{split}
	\end{equation*}
\end{thm}

\section{The Diamond ensemble}\label{diamond}

We are now ready to define the construction that leads to Theorem \ref{ThmPpal2}.
It amounts to choose some $r_1,\ldots,r_p$ such that the energy bound computed in Theorem \ref{cor:nuevasuma} is as low as possible and can be computed up to order $o(N)$.
Our construction is based in the following heuristic argument.
\subsection{A heuristic argument}
Let us choose $z_1,\ldots,z_p$ in such a way that they define $p$ equidistant (for the spherical distance) parallels on the sphere. In other words,
\[
z_j=\cos\frac{j}{p+1}
\]
The distance between two consecutive parallels is $\pi/(p+1)$.
We would like to choose $r_j$ in such a way that the distance between two consecutive points of the same parallel is approximately equal to some constant times $\pi/(p+1)$. Since parallel of height $z_j$ is a circumference of radius $\sin(j/(p+1))$, this goal is attained by setting for example
\begin{equation}\label{eq:ideal}
r_{j}
=
\frac{K_0\pi\sin \left( \frac{j\pi}{p+1} \right)}{\sin \left( \frac{\pi}{2(p+1)} \right)}
.
\end{equation}
Let us forget for a moment that this gives an impossible construction (since the $r_j$ will not be integer numbers). One can then plug in in Proposition \ref{PropEnergyOmegaGeneral} these values of $z_j$ and $r_j$. After a considerable amount of work the right--hand term in Proposition \ref{PropEnergyOmegaGeneral} can be proved to have the asymptotic expansion
\begin{equation}\label{eq:auxideal}
\begin{split}
W_{\log}(\S2)N^2
-\frac{1}{2} N \log(N)
+ \left(\frac{K_0\pi}{6}-\frac{1}{2}\log K_0-\frac{\log\pi}{2}\right)N
+ \mathfrak{o}(N),
\end{split}
\end{equation}
where $N=2+r_1+\cdots+r_p$ is the total number of points in the sphere. The optimal value of $K_0$ is $K_0=3/\pi$, yielding the asymptotic
\begin{equation*}
\begin{split}
&
\Esp{\theta_{1},...\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega)}
\leq
- \frac{\log\left( \frac{4}{e} \right)}{2} N^2
-\frac{1}{2} N \log(N)
+ \frac{1-\log(3)}{2} N
+ \mathfrak{o}(N),
\end{split}
\end{equation*}
where $\frac{1-\log(3)}{2} \approx -0.0493$.
\medskip

Unfortunately, this reasoning does not actually produce collections of points since as pointed out above the number of points in each parallel must be an integer number. The computation of the formula \eqref{eq:auxideal} is done with techniques similar to the ones used below but we do not include it since we actually only use it as an inspiration of our true construction below.
\subsection{An actual construction}
Inspired on the heuristic argument above, we will try to search for sets of the form $\Omega(p,r_j)$ such that the $r_{j}$ are integer numbers close to $\frac{3\sin \left( \frac{j\pi}{p+1} \right)}{\sin \left( \frac{\pi}{2(p+1)} \right)}$. We will then choose the optimal values for the $z_j$ given by Proposition \ref{Propminparallel}. Our approach is to consider different piecewise linear approximations to the formula \eqref{eq:ideal} with $K_0=3/\pi$.

\begin{defn}
Let $p,M$ be two positive integers with $p=2M-1$ odd and let $r_{j}=r(j)$ where $r:[0,2M]\to\mathbb{R}$ is a continuous piecewise linear function satisfying $r(x)=r(2M-x)$ and
\begin{equation*}
r(x)
=
   \begin{cases}
      \alpha_1+\beta_1 x              & \mbox{if } 0=t_0 \leq x \leq t_1   \\
      \vdots&\vdots\\
\alpha_n+\beta_n x              & \mbox{if } t_{n-1} \leq x \leq t_n=M
   \end{cases}
\end{equation*}
Here, $[t_0,t_1,\ldots,t_n]$ is some partition of $[0,M]$ and all the $t_\ell, \alpha_\ell,\beta_\ell$ are assumed to be integer numbers.

We assume that $\alpha_1=0$, $\alpha_\ell,\beta_\ell\geq0$ and $\beta_1>0$ and there exists a constant $A\geq2$ not depending on $M$ such that $\alpha_\ell\leq AM$ and $\beta_\ell\leq A$. We also assume that $t_1\geq cM$ for some $c\geq0$. Moreover, let $z_{j}$ be as defined in Proposition \ref{Propminparallel}.

We call the set of points defined this way the \textit{Diamond ensemble} and we denote it by $\diamond (N)$, omiting in the notation the dependence on all the parameters $n$, $t_1,\ldots,t_n$, $\alpha_1,\ldots,\alpha_n$, $\beta_1,\ldots,\beta_n$. Note that the total number of points is
\[
N=2-(\alpha_n+\beta_nM)+2\sum_{\ell=1}^n\;\sum_{j=t_{\ell-1}+1}^{t_\ell}(\alpha_\ell+\beta_\ell j).
\]
We also denote by $N_\ell$ the total number of points in up to $t_{\ell-1}$, that is
\[
N_\ell=\sum_{j=1}^{t_{\ell-1}-1}r_j.
\]
Note that if $j\in[t_{\ell-1},t_\ell]$ then
\begin{multline}\label{zjs}
z_j=1-\frac{1+r_j+2\sum_{k=1}^{j-1}r_k}{N-1}=1-\frac{1+2N_j-r_j+2\sum_{k=t_{\ell-1}}^{j}(\alpha_\ell+\beta_\ell k)}{N-1}\\
=1-\frac{1+2N_j-(\alpha_\ell+\beta_\ell j)+2\alpha_\ell(j-t_{\ell-1}+1)+\beta_\ell(j+t_{\ell-1})(j-t_{\ell-1}+1)}{N-1}
\end{multline}
We thus consider the function $z(x)$ piecewise defined by the degree $2$ polynomial
\begin{multline}\label{zjs2}
z_\ell(x)=1-\frac{1+2N_j-(\alpha_\ell+\beta_\ell x)+2\alpha_\ell(x-t_{\ell-1}+1)+\beta_\ell(x+t_{\ell-1})(x-t_{\ell-1}+1)}{N-1}
\end{multline}
\end{defn}
and note that $z_j=z(j)$.
\subsection{An exact formula for the expected logarithmic energy of the Diamond ensemble}
From Theorem \ref{cor:nuevasuma}, the expected value of the log-energy of $\diamond(N)$ is given by
	\begin{equation*}
	\begin{split}
	&
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond(N))}
	\\
	& =
	-(N-1)\log(4)+r(M)\log r(M)-2\sum_{j=1}^{M}r(j)\log r(j)\\&-(N-1)\sum_{j=1}^Mr(j)(1-z(j))\log(1-z(j))-(N-1)\sum_{j=1}^Mr(j)(1+z(j))\log(1+z(j)).
	\end{split}
	\end{equation*}
We write the sums as instances of a trapezoidal composite rule. Recall that for a function  $f:[a,b]\to\mathbb{R}$ with $a<b$ integers, the composite trapezoidal rule is
\begin{equation}\label{eq:int_sumprevio}
T_{[a,b]}(f)=\frac{f(a)+f(b)}{2}+\sum_{j=a+1}^{b-1} f(j).
\end{equation}
We then have
\begin{cor}\label{cor:otrasuma}
The expected logarithmic energy of points drawn from the Diamond ensemble equals
		\begin{equation*}
		\begin{split}
		&
		\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond(N))} =-(N-1)\log(4)-2\sum_{\ell=1}^nT_{[t_{\ell-1},t_\ell]}(f_\ell)
		\\&-(N-1)\sum_{\ell=1}^nT_{[t_{\ell-1},t_\ell]}(g_\ell)-(N-1)\sum_{\ell=1}^nT_{[t_{\ell-1},t_\ell]}(h_\ell),
		\end{split}
		\end{equation*}
		where for $1\leq \ell\leq n$ the functions $f_\ell,g_\ell,h_\ell$ are defined in the interval $[t_\ell-1,t_\ell]$ and satisfy
		\begin{align*}
		f_\ell(x)=&(\alpha_\ell+\beta_\ell x)\log(\alpha_\ell+\beta_\ell x)\\
		g_\ell(x)=&(\alpha_\ell+\beta_\ell x)(1-z_\ell(x))\log(1-z_\ell(x))\\
		h_\ell(x)=&(\alpha_\ell+\beta_\ell x)\left(1+z_\ell(x)\right)\log(1+z_\ell(x))\\
		\end{align*}
\end{cor}
\subsection{An asymptotic formula for the expected logarithmic energy of the Diamond ensemble}
Since $f_\ell$ is a continuous function for $1\leq\ell\leq n$, the trapezoidal rule $T_{[t_{\ell-1},t_\ell]}(f_\ell)$ approaches the integral of $f_\ell$. Moreover,
\begin{lem}\label{lem:integralfacil}
	For $1\leq \ell\leq n$ we have
	\[
	\left|T_{[t_{\ell-1},t_\ell]}(f_\ell)-\int_{t_{{\ell-1}}}^{t_\ell}f_\ell(x)\,dx\right|\leq
	(t_\ell-t_{\ell-1})3K\log(2KM)\leq 3KM\log(2KM).\]
\end{lem}
\begin{proof}
	Let $S$ be the quantity in the lemma and note that
	\[
	S\leq\sum_{j=t_{\ell-1}+1}^{t_\ell}\int_{j-1}^j\left|f_\ell(x)-\frac{f_\ell(j-1)+f_\ell(j)}{2}\right|\,dx.
	\]
	Now, for $x\in[j-1,j]$ we have
	\[
	|f_\ell(x)-f_\ell(j-1)|\leq\int_{j-1}^j|f_\ell'(t)|\,dt\leq 1+K\log(KM+KM)\leq 2K\log(2KM).
	\]
	We thus have
	\begin{multline*}
	\left|f_\ell(x)-\frac{f_\ell(j-1)+f_\ell(j)}{2}\right|\leq\\\left|f_\ell(x)-f_\ell(j-1)\right|+\left|\frac{f_\ell(j-1)-f_\ell(j)}{2}\right|\leq 3K\log(2KM).
	\end{multline*}
	The lemma follows.
\end{proof}
Indeed, we can use the classical Euler-Maclaurin formula (see for example \cite[Th. 9.26]{Kress}) for estimating the difference between the composite trapezoidal rule and the integral in the cases of $g_\ell$ and $h_\ell$. Indeed we have
\begin{lem}\label{lem:eulermclaurin}
	The following inequality holds for $1\leq \ell\leq n$:
	\begin{align*}
	\left|T_{[t_{\ell-1},t_\ell]}(g_\ell)-\int_{t_\ell-1}^{t_\ell}g_\ell(x)\,dx-\frac{g_\ell'(t_\ell)-g_\ell'(t_{\ell-1})}{12}\right|\leq\frac{C\log M}{M}
	\end{align*}
	for some constant $C>0$.
\end{lem}
\begin{proof}
 From Lemma \ref{lem:em} it suffices to  prove that $|g_\ell'''(x)|\,dx\leq \frac{C\log M}{M}$ for some constant $C$. Now, $g_\ell=u(x)v(x)w(x)$ where $u$ is a linear mapping, $v$ is a quadratic polynomial and $w=\log v$. The Leibniz rule for the derivative of the product gives
\[
g_\ell'''=uvw'''+6u'v'w'+3u'v''w+3uv''w'+3u'vw''+3uv'w''.
\]
If $\ell=1$ then $g_\ell'''$ has a simple expression and it is easily verified that
\[
g_1'''\leq\frac{C\log M}{M^2}
\]
for some constant $C>0$. For $\ell>1$ note now that $u(x)=\alpha_\ell +\beta_\ell x$ satisfies
\[
|u|\leq CM,\quad |u'|\leq C
\]
where $C$ is some constant. Moreover, $v(x)=1-z_\ell(x)$ satisfies
\[
c\leq |v|<1,\quad |v'|\leq \frac{C}{M},\quad |v''|\leq\frac{C}{M^2}
\]
for some positive constant $c$, not depending on $M$.
Also, $w=\log v$ satisfies
\[
|w|\leq C,\quad|w'|\leq \frac{C}{M},\quad |w''|\leq \frac{C}{M^2}\quad |w'''|\leq \frac{C}{M^3}.
\]
The lemma follows.
\end{proof}

\begin{lem}\label{lem:eulermclaurin2}
	The following inequality holds for $1\leq \ell\leq n$:
	\begin{align*}
	\left|T_{[t_{\ell-1},t_\ell]}(h_\ell)-\int_{t_\ell-1}^{t_\ell}h_\ell(x)\,dx-\frac{h_\ell'(t_\ell)-h_\ell'(t_{\ell-1})}{12}\right|\leq \frac{C}{M}
	\end{align*}
	for some constant $C>0$.
\end{lem}
\begin{proof}
The proof is almost identical to that of Lemma \ref{lem:eulermclaurin}, so we leave it to the reader.
\end{proof}
We have proved the following.
\begin{thm}\label{maintechnical}
For the Diamond ensemble we have
		\begin{equation*}
		\begin{split}
		&
		\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond(N))} =-(N-1)\log(4)-2\sum_{\ell=1}^n\int_{t_{{\ell-1}}}^{t_\ell}f_\ell(x)\,dx
		\\&-(N-1)\sum_{\ell=1}^n\left(\int_{t_\ell-1}^{t_\ell}g_\ell(x)\,dx+\frac{g_\ell'(t_\ell)-g_\ell'(t_{\ell-1})}{12}\right)\\&-(N-1)\sum_{\ell=1}^n\left(\int_{t_\ell-1}^{t_\ell}h_\ell(x)\,dx+\frac{h_\ell'(t_\ell)-h_\ell'(t_{\ell-1})}{12}\right)+o(M^2),
		\end{split}
		\end{equation*}
		where as before for $1\leq \ell\leq n$ the functions $f_\ell,g_\ell,h_\ell$ are as in Corollary \ref{cor:otrasuma}.
\end{thm}

\subsubsection{Zonal Equal Area Nodes}

In \cite{RSZ94} Rakhmanov et al. define a diameter bounded, equal area partition of $\mathbb{S}^{2}$ consisting on two spherical caps on the south and the north pole and rectiliniar cells located on rings of parallels.
The resemblance between our model and this model is remarkable, and even if the
constructions are different, the points obtained seem to be really close.
Actually, both the authors in \cite{RSZ94} and ourselves try to approximate $r_{j}$ as in equation \eqref{eq:ideal} by an integer number.
The theoretical bounds we obtain here for the logarithmic energy are slightly better than the numerical bounds obtained in \cite{dolomites} for the zonal equal area nodes.

An interesting fact is that among all the algorithmically generated point sets, the generalized spiral and zonal equal area points perform the best with respect to the logarithmic energy.
\cite[Proposition 2.3.]{dolomites} claims that the  sequence of zonal equal area configurations is equidistributed and quasi-uniform.
The same kind of result can probably be stated for the Diamond ensemble.

\section{Concrete examples of the Diamond ensemble}\label{examples}

Throughout this section we are going to explore three different examples of the Diamond ensemble.
Each of them is ilustrated with two kinds of figures: a concrete example of points following the model on $\mathbb{S}^{2}$ (figures \ref{fig_sphere_simp_01}, \ref{fig_sphere_elab_01} and \ref{fig_sphere_quasi_01}) and a comparative beetwen the $r_{j}$ that define the model and the $r_{j}$ in equation \eqref{eq:ideal} with $K_{0}=3/\pi$ (figures \ref{fig_simp_02}, \ref{fig_elab_02} and \ref{fig_quasi_02}).
In figures \ref{fig_sphere_simp_01}, \ref{fig_sphere_elab_01} and \ref{fig_sphere_quasi_01} we have used different colors for points obtained from the different linear pieces defining $r(x)$.

\subsection{A simple example}\label{sec:simple}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{sphere_simp_colors.png}
\caption{A realization of a simple example with $K=4$ and $N=1602$.} \label{fig_sphere_simp_01}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{r_j_simp.png}
\caption{Number of points for a simple example with $K=4$.}\label{fig_simp_02}
\end{figure}

We choose $n=1$, $r_j=Kj$ with $K$ a positive integer for $1\leq j\leq M$. Then, for $l\in\{1,\ldots,M\}$ we have
\[
z_l=1-\frac{1+Kl^2}{N-1}.
\]
The number of parallels is $2M-1$ and the number of points of the Diamond ensemble is
\begin{equation*}
N
=
2 + \sum_{j=1}^{p} r_{j}
=
2 -KM+ 2\sum_{j=1}^{M} Kj
=
2 + KM^2.
\end{equation*}

One can then write down the functions in Corollary \ref{cor:otrasuma} getting
\begin{align*}
f(x)=&Kx\log(Kx).\\
g(x)=&Kx\frac{1+Kx^2}{N-1}\log\frac{1+Kx^2}{N-1}.\\
h(x)=&Kx\left(2-\frac{1+Kx^2}{N-1}\right)\log\left(2-\frac{1+Kx^2}{N-1}\right).\\
\end{align*}
Then, the formula in Theorem \ref{maintechnical} reads
\begin{equation*}
\begin{split}
&
\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond(N))} =-(N-1)\log(4)-2\int_{0}^{M}f(x)\,dx
\\&-(N-1)\left(\int_{0}^{M}g(x)+h(x)\,dx+\frac{g'(M)-g(0)}{12}+\frac{h'(M)-h'(0)}{12}\right)+o(M^2).
\end{split}
\end{equation*}
All these integrals and derivatives can be computed, obtaining the following result.

\begin{thm}\label{thm:mainsimple}

	The expected value of the logarithmic energy of the Diamond ensemble in this section is
	\begin{multline*}
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond (N))}
	=W_{\log}(\S2)N^2-\frac{1}{2}N\log N\\+N\left(\frac{\log 2}{6}K-\frac12+\log 2-\frac{\log K}{2}\right)+o(N).
	\end{multline*}
	In particular, if $K=4$ we have
	\begin{equation*}
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond (N))}
	=W_{\log}(\S2)-\frac{1}{2}N\log N+N\left(\frac{2\log 2}{3}-\frac12\right)+o(N).
	\end{equation*}
\end{thm}
Note that $\frac{2\log 2}{3}-\frac12=- 0.037901879626703\ldots$
Using this simple example we are thus approximately $0.0177$ far from the valued conjectured in \eqref{conjetura}.

\subsubsection{Octahedral configurations of points}
In \cite{HOLHOS20141092} an area preserving map from the unit sphere to the regular octahedron is defined.
Considering some hierarchical triangular grids on the facets of the octahedron a grid can be mapped into the sphere obtaining two different sets of points: those coming from the vertex of the grid $\Omega_{N}$ and the centers of the triangles $\Lambda_{N}$.

$\Omega_{N}$ consists on $4M^ 2 + 2$ points in the sphere that are a concrete example (with fixed angles) of our simple example.
In the paper, the authors give some numerical simulations for the logarithmic energy of this set of points that are confirmed by Theorem \ref{thm:mainsimple}.
Also in \cite[Figure 2.2]{dolomites} new numerical simulations for the same set are done obtaining a bound which is very similar to the one we prove here.

\subsection{A more elaborated example}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{sphere_elab_colors_m_4.png}
\caption{A realization of a more elaborated example with $N=1314$.} \label{fig_sphere_elab_01}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{r_j_elab.png}
\caption{Number of points for a more elaborated example.} \label{fig_elab_02}
\end{figure}

The following choice of $r_j$ produces much better results. Let $p=2M-1$ where $M=4m$ with $m$ a positive integer. Let $n=3$ and let $r_j=r(j)$ where
\[
r(x)=
\begin{cases}
6x&0\leq x\leq 2m
\\6m+3x&2m\leq x\leq 3m
\\12m+x&3m\leq x\leq 4m
\\20m-x&4m\leq x\leq 5m
\\30m-3x&5m\leq x\leq 6m
\\48m-6x&6m\leq x\leq 8m
\end{cases}
\]
that satisfies $r(x)=r(p+1-x)=r(8m-x)$. Let $z_j=z(j)$ where $z(x)$ is defined by \eqref{zjs2}, that is,

\[
z(x)=
\begin{cases}
\frac{82m^2-6x^2}{82m^2+1}&0\leq x\leq 2m
\\\frac{94m^2-12mx-3x^2}{82m^2+1}&2m\leq x\leq 3m
\\\frac{112m^2-24mx-x^2}{82m^2+1}&3m\leq x\leq 4m
\\\frac{144m^2-40mx+x^2}{82m^2+1}&4m\leq x\leq 5m
\\\frac{194m^2-60mx+3x^2}{82m^2+1}&5m\leq x\leq 6m
\\\frac{302m^2-96mx+6x^2}{82m^2+1}&6m\leq x\leq 8m
\end{cases}
\]
We moreover have $N=82m^2+2$. Again, all the integrals and derivatives in Theorem \ref{maintechnical} can be computed, although this time the computer algebra package {\tt Maxima} has been used, getting the following result.

\begin{thm}\label{thm:main}

The expected value of the logarithmic energy of the Diamond ensemble in this section is
	\begin{equation*}
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\diamond (N))}
	=W_{\log}(\S{2})N^2-\frac{N}{2}\log N+cN+o(N),
	\end{equation*}
	where $c=- 0.048033870622806\ldots$ satisfies
	\begin{multline*}
	492c=- 113 \log 113 - 982 \log82 - 210 \log70 - 51 \log51\\
	+ 1638 \log41 + 900 \log15 - 36 \log12 - 1536 \log8 \\+ 144 \log6
	- 492 \log4 + 1968 \log2 - 246.
	\end{multline*}
\end{thm}
Using this more elaborated example we are thus approximately $0.0076$ far from the value conjectured in \eqref{conjetura}.

\subsection{A quasioptimal Diamond example}\label{sec:qode}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{sphere_quasi_colors.png}
\caption{A realization of a quasioptimal example with $N=958$.} \label{fig_sphere_quasi_01}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{r_j_quasi.png}
\caption{A realization of a quasioptimal example.}\label{fig_quasi_02}
\end{figure}

We have made a number of tries with different choices of the parameters for the Diamond ensemble. The best one (i.e. the one with minimal logarithmic energy) that we have found is the following one: let $M=7m$ with $m$ a positive integer, let $p=2M-1$ and let
\[
r(x)=
\begin{cases}
6x&0\leq x\leq 2m
\\2m+5x&2m\leq x\leq 3m
\\5m+4x&3m\leq x\leq 4m
\\9m+3x&4m\leq x\leq 5m
\\14m+2x&5m\leq x\leq 6m
\\20m+x&6m\leq x\leq 7m
\\34m-x&7m\leq x\leq 8m
\\42m-2x&8m\leq x\leq 9m
\\51m-3x&9m\leq x\leq 10m
\\61m-4x&10m\leq x\leq 11m
\\72m-5x&11m\leq x\leq 12m
\\84m-6x&12m\leq x\leq 14m=p+1
\end{cases}
\]
that satisfies $r(x)=r(p+1-x)=r(14m-x)$. Let $z_j=z(j)$ where $z(x)$ is defined by \eqref{zjs2}, that is,
\[
z(x)=
\frac{1}{239m^2+1}\times\begin{cases}
{239m^2-6x^2}&0\leq x\leq 2m
\\{243m^2-4mx-5x^2}&2m\leq x\leq 3m
\\{252m^2-10mx-4x^2}&3m\leq x\leq 4m
\\{268m^2-18mx-3x^2}&4m\leq x\leq 5m
\\{293m^2-28mx-2x^2}&5m\leq x\leq 6m
\\{329m^2-40mx-x^2}&6m\leq x\leq 7m
\\{427m^2-68mx+x^2}&7m\leq x\leq 8m
\\{491m^2-84mx+2x^2}&8m\leq x\leq 9m
\\{572m^2-102mx+3x^2}&9m\leq x\leq 10m
\\{672m^2-122mx+4x^2}&10m\leq x\leq 11m
\\{793m^2-144mx+5x^2}&11m\leq x\leq 12m
\\{937m^2-168mx+6x^2}&12m\leq x\leq 14m=p+1
\end{cases}
\]
We moreover have $N=239m^2+2$. Again, all these integrals and derivatives have been computed by the computer algebra package {\tt Maxima}, obtaining Theorem \ref{ThmPpal2}.

\begin{figure}[h]
\centering
\includegraphics[scale=0.7]{r_j_simp_elab_quasi.png}
\label{fig_r_j_simp_elab_quasi}
\caption{Comparison of the number of points in each parallel for the different models for $2M \approx 2.500.000$ parallels.}
\end{figure}

\section{Proofs of the main results}\label{section_proofs}

\subsection{Proof of Proposition \ref{PropAveEner}}

In order to prove Proposition \ref{PropAveEner}, we will need the following Lemma.

\begin{lem}\cite[Formula 4.224]{zwillinger2014table} \label{LemInt1}
	The equation
	\begin{equation*}
	\int_{0}^{\pi} \log(a + b\cos(\theta)) d\theta
	=
	\pi\log \left( \frac{a + \sqrt{a^{2} - b^{2}}}{2} \right)
	\end{equation*}
	is satisfied if $a \geq |b| > 0$.
\end{lem}

\begin{dem}
Note that
	\begin{equation*}
	\begin{split}
	& ||x-y||
	=
	\left|\left|
	\left( \sqrt{1 - z_{i}^{2}} \cos \theta_{i}, \sqrt{1 - z_{i}^{2}} \sin \theta_{i}, z_{i} \right)
	-
	\left( \sqrt{1 - z_{j}^{2}} \cos \theta_{j}, \sqrt{1 - z_{j}^{2}} \sin \theta_{j}, z_{j} \right)
	\right|\right|
	\\
	& =
	\sqrt{2}\sqrt{1 -z_{i}z_{j} - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}} \cos(\theta_{i} - \theta_{j})}.
	\end{split}
	\end{equation*}

	\noindent We compute then
	\begin{equation*}
	\begin{split}
	& \Esp{\theta_{i},\theta_{2}}{-\log\left( \|x-y\| \right)}
	\\
	& =
	\frac{1}{4\pi^{2}}
	\int_{0}^{2\pi} \int_{0}^{2\pi}
	- \log \left(
	\sqrt{2}\sqrt{1 -z_{i}z_{j} - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}} \cos(\theta_{i} - \theta_{j})}
	\right) d\theta_{i} d\theta_{j}
	\\
	\end{split}
	\end{equation*}
	\begin{equation*}
	\begin{split}
	& =
	\frac{-\log(2)}{2}
	- \frac{1}{8\pi^{2}}
	\int_{0}^{2\pi} \int_{0}^{2\pi}
	\log \left(
	1 -z_{i}z_{j} - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}} \cos(\theta_{i} - \theta_{j})
	\right) d\theta_{i} d\theta_{j}
	\\
	& =
	\frac{-\log(2)}{2}
	- \frac{1}{4\pi}
	\int_{0}^{2\pi}
	\log \left(
	1 -z_{i}z_{j} - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}} \cos(\theta)
	\right) d\theta
	\\
	& =
	\frac{-\log(2)}{2}
	- \frac{1}{2\pi}
	\int_{0}^{\pi}
	\log \left(
	1 -z_{i}z_{j} - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}} \cos(\theta)
	\right) d\theta.
	\end{split}
	\end{equation*}

	\noindent From Lemma \ref{LemInt1} with $a = 1 -z_{i}z_{j}$, $b = - \sqrt{1 - z_{i}^{2}} \sqrt{1 - z_{j}^{2}}$ and $z_{i} \neq z_{j}$ we have that
	\begin{equation*}
	\begin{split}
	\Esp{\theta_{i},\theta_{2}}{-\log\left( \|x-y\| \right)}
	& =
	\frac{-\log(2)}{2}
	- \frac{1}{2\pi}
	\pi
	\log \left( \frac{1 - z_{i}z_{j} + |z_{i} - z_{j}|}{2} \right)
	\\
	& =
	\frac{-\log(2)}{2}
	- \frac{1}{2}
	\left[
	\log \left( 1 - z_{i}z_{j} + |z_{i} - z_{j}| \right)
	-\log(2)
	\right]
	\\
	& =
	- \frac{\log \left( 1 - z_{i}z_{j} + |z_{i} - z_{j}| \right)}{2}.
	\end{split}
	\end{equation*}

	$\square$

\end{dem}

\subsection{Proof of Proposition \ref{PropEnergyOmegaGeneral}}

In order to compute de logarithmic energy associated to $\Omega(p,r_{j},z_{j})$, we have to sum the following quantities:

\begin{itemize}
	\item $A$: the energy between each point $x_{j}^{i}$, $1\leq j \leq p$ and $1\leq i \leq r_{j}$ and the north and the south pole, counted twice and the energy from the south to the north pole, again counted twice.
	\item $B$: the energy of the scaled roots of unity for every parallel $1 \leq j \leq p$.
	\item $C$: the energy between the points of every pair of parallels, as in Corollary \ref{CorAveEner2}.
\end{itemize}

\subsubsection{Computation of quantity $A$}

Note that
\begin{equation*}
\begin{split}
\|(0,0,1)-x_j^i\|=&
\sqrt{2}\sqrt{1 - z_{j}},\\
\|(0,0,-1)-x_j^i\|=&
\sqrt{2}\sqrt{1 + z_{j}}.
\end{split}
\end{equation*}

Quantity $A$ thus equals
\begin{equation}\label{QuantityA}
\begin{split}
A=&
-2\log(2)
-\sum_{j=1}^{p} r_{j} \left( \log(4) + \log \left(  1 - z_{j}^{2}  \right)
\right).
\end{split}
\end{equation}

\subsubsection{Computation of quantity $B$}

We will use the following results from \cite[Pg. 3]{BLMS:BLMS0621}: the logarithmic energy  associated to $N$ roots of unity in the unit circumference is $-N\log N$. As a trivial consequence, the logarithmic energy associated to $N$ points which are equidistributed in a circumference of radius $R$ is $-N\log N-N(N-1)\log R$.

Since the parallel at height $z_j$ is a circumference of radius $\sqrt{1-z_j^2}$, quantity $B$ equals

\begin{equation}\label{QuantityB}
\begin{split}
B& =
-\sum_{j=1}^{p}
r_{j}\log r_{j} + \frac{r_{j}(r_{j} - 1)}{2} \log (1 - z_{j}^{2}).
\end{split}
\end{equation}

\subsubsection{Computation of quantity $C$}

This has been done in Corollary \ref{CorAveEner2}:

\begin{equation}\label{QuantityC}
C= \sum_{k,j=1; k\neq j}^{p}  - r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}.
\end{equation}

In order to compute the logarithmic energy associated to the set $\Omega(p,r_{j},z_{j})$ it only rest to sum the quantities \eqref{QuantityA}, \eqref{QuantityB} and \eqref{QuantityC}.

\begin{equation*}
\begin{split}
&
\Esp{\theta_{1},...\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j},z_{j}))}
=
-2\log(2)
-\sum_{j=1}^{p} r_{j} \left( \log(4) + \log \left(  1 - z_{j}^{2}  \right)
\right)
\\
&  -
\sum_{j=1}^{p} \left(r_{j}\log r_{j} + \frac{r_{j}(r_{j} - 1)}{2} \log (1 - z_{j}^{2})\right)
-
\sum_{j=1}^{p} \sum_{k\neq j}  r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}
\\
= &
-2\log(2)
-\sum_{j=1}^{p}
\left[
r_{j}\log(4)
+ r_{j}\log(1 - z_{j}^{2})
+ r_{j}\log r_{j}
+ \frac{r_{j}^{2}}{2} \log (1- z_{j}^{2})
\right.
\\
& \left.
- \frac{r_{j}}{2} \log (1- z_{j}^{2})
+ \sum_{k\neq j}  r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}
\right]
\\
= &
-2\log(2)
-\sum_{j=1}^{p}
\left[
r_{j}\log(4)
+ \frac{r_{j}}{2}  \log(1 - z_{j}^{2})
+ r_{j}\log r_{j}
+ \sum_{k=1}^{p}  r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}
\right].
\end{split}
\end{equation*}

$\square$

\subsection{Proof of Proposition \ref{Propminparallel}}

We derivate the formula from Proposition \ref{PropEnergyOmegaGeneral} for $z_{l}$ obtaining:

\begin{equation*}
\begin{split}
&
\frac{\partial \Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j},z_{j}))}
}{\partial z_{l}}
\\
& =
\frac{\partial}{\partial z_{l}}
\left(
-\sum_{j=1}^{p}
\frac{r_{j}}{2}  \log(1 - z_{j}^{2})
-\sum_{j=1}^{p}\sum_{k=1}^{p}
r_{j}r_{k}\frac{\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)}{2}
\right) \\
& =
\frac{z_{l}r_{l}}{1-z_{l}^{2}}
+ \frac{z_{l}r_{l}^{2}}{1-z_{l}^{2}}
+ \sum_{j=1}^{l-1} r_{j}r_{l} \frac{1+z_{j}}{(1-z_{l})(1+z_{j})}
- \sum_{j=l+1}^{p} r_{j}r_{l} \frac{1-z_{j}}{(1+z_{l})(1-z_{j})}\\
& =
\frac{z_{l}r_{l}(1+r_{l})}{1-z_{l}^{2}}
+ \sum_{j=1}^{l-1}  \frac{r_{j}r_{l}}{1-z_{l}}
- \sum_{j=l+1}^{p}  \frac{r_{j}r_{l}}{1+z_{l}} \\
& =
\frac{r_{l}}{1-z_{l}^{2}}
\left(
(1+r_{l})z_{l}
+ (1+z_{l})\sum_{j=1}^{l-1}  r_{j}
- (1-z_{l})\sum_{j=l+1}^{p}  r_{j}
\right) \\
& =
\frac{r_{l}}{1-z_{l}^{2}}
\left(
z_{l}
+ \sum_{j=1}^{l-1}  r_{j}
- \sum_{j=l+1}^{p}  r_{j}
+ z_{l} \sum_{j=1}^{p}  r_{j}
\right)
.
\end{split}
\end{equation*}

We have then
\begin{equation*}
\begin{split}
&
\frac{\partial \Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j},z_{j}))}
}{\partial z_{l}}
=
0
\iff
z_{l}
\left(
1 + \sum_{j=1}^{p}  r_{j}
\right)
=
\sum_{j=l+1}^{p}  r_{j}
-
\sum_{j=1}^{l-1}  r_{j}
.
\end{split}
\end{equation*}

In other words,
\begin{equation*}
z_{l}
=
\frac{\displaystyle\sum_{j=l+1}^{p}r_{j} - \displaystyle\sum_{j=1}^{l-1}r_{j}}{1 + \displaystyle\sum_{j=1}^{p}r_{j}}.
\end{equation*}

$\square$

\subsection{Proof of Theorem \ref{cor:nuevasuma}}

To prove Theorem \ref{cor:nuevasuma} the following lemma will be useful.

\begin{lem}\label{lem3}
If $r_j=r_{p+1-j}$ and $z_j$ are chosen as in Proposition \ref{Propminparallel} we then have
	\begin{align*}
	\frac{1}{2}\sum_{j=1}^{p} \sum_{k=1}^{p}  r_{j}r_{k}\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)=&(N-1)\sum_{j=1}^{p} r_{j}(1-z_j)\log(1 - z_{j})\\-&\sum_{j=1}^{p} r_{j}\log(1 - z_j).
	\end{align*}
\end{lem}
\begin{proof}
	Let
	\[
	a_{j,k}=r_{j}r_{k}\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right) ,\quad 	b_{j,k}=r_{j}r_{k}\log \left( 1 + z_{j}z_{k} + |z_{j} + z_{k }| \right)
	\]
	and note that they satisfy:
	\[
	a_{j,k}=a_{k,j},\quad a_{j,p+1-k}=b_{j,k} ,\quad a_{p+1-j,k}=b_{j,k} ,\quad a_{p+1-j,p+1-k}=a_{j,k},\quad a_{M,M}=0.
	\]
	We thus have
	\[
	\sum_{j=1}^{p} \sum_{k=1}^{p}a_{j,k}=\sum_{j=1}^pa_{j,j}+\sum_{\stackrel{j,k=1}{j\neq k}}^{p} a_{j,k}=2\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2)+\sum_{\stackrel{j,k=1}{j\neq k}}^{p} a_{j,k}.
	\]
	Moreover, recalling that $p=2M-1$,
	\begin{align}\label{eq:unaporaqui}
	\sum_{\stackrel{j,k=1}{j\neq k}}^{p} a_{j,k}=&\sum_{\stackrel{j,k=1}{j\neq k}}^Ma_{j,k}+\sum_{j=1}^{M} \sum_{k=M+1}^{2M-1}a_{j,k}+\sum_{j=M+1}^{2M-1} \sum_{k=1}^{M}a_{j,k}+\sum_{\stackrel{j,k=M+1}{j\neq k}}^{2M-1}a_{j,k}.
	\end{align}
	The two sums in the middle  of the right hand term in \eqref{eq:unaporaqui} can be rewritten as
	\[
	\sum_{j=1}^{M} \sum_{k=1}^{M-1}b_{j,k}+\sum_{j=1}^{M-1} \sum_{k=1}^{M}b_{j,k}=2\sum_{j=1}^{M} \sum_{k=1}^{M-1}b_{j,k},
	\]
	and using that $z_j\geq0$ for $1\leq j\leq M$ this last equals
	\begin{align*}
	2\sum_{j=1}^{M} \sum_{k=1}^{M-1}b_{j,k}=&	2\sum_{j=1}^{M} \sum_{k=1}^{M-1}r_jr_k\log(1+z_j)+	2\sum_{j=1}^{M} \sum_{k=1}^{M-1}r_jr_k\log(1+z_k)\\
	=&2\left(\sum_{k=1}^{M-1}r_k\right)\sum_{j=1}^{M} r_j\log(1+z_j)+	2\left(\sum_{j=1}^{M}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k)\\
	=&2\left(r_M+2\sum_{j=1}^{M-1}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k),
	\end{align*}
	where in the last step we have used that $z_M=0$. From \eqref{eq:unaporaqui} we then have proved that the sum in the lemma equals
	\begin{multline*}
	\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2)+\left(r_M+2\sum_{j=1}^{M-1}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k)+\frac12\sum_{\stackrel{j,k=1}{j\neq k}}^Ma_{j,k}+\frac12\sum_{\stackrel{j,k=M+1}{j\neq k}}^{2M-1}a_{j,k}=\\	\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2)+\left(r_M+2\sum_{j=1}^{M-1}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k)+\frac12\sum_{\stackrel{j,k=1}{j\neq k}}^Ma_{j,k}+\frac12\sum_{\stackrel{j,k=1}{j\neq k}}^{M-1}a_{j,k},
	\end{multline*}
	where we have used $a_{p+1-j,p+1-k}=a_{j,k}$. The two sums in the expression above have many common terms. We can rearrange them as follows:
	\begin{align*}
	\frac12\sum_{\stackrel{j,k=1}{j\neq k}}^Ma_{j,k}+\frac12\sum_{\stackrel{j,k=1}{j\neq k}}^{M-1}a_{j,k}=&\sum_{\stackrel{j,k=1}{j\neq k}}^{M-1}a_{j,k}+\frac12\sum_{k=1}^{M-1}a_{M,k}+\frac12\sum_{j=1}^{M-1}a_{j,M}\\
	=&2\sum_{j=1}^{M-1}\sum_{k=1}^{j-1}a_{j,k}+\sum_{k=1}^{M-1}a_{M,k}\\
	=&2\sum_{j=1}^{M-1}\sum_{k=1}^{j-1}a_{j,k}+\sum_{k=1}^{M-1}r_{M}r_{k}\log \left( 1+ z_{k } \right) ,
	\end{align*}
	where again we are using $a_{j,k}=a_{k,j}$ and $z_M=0$. All in one, we have proved that the sum in the lemma equals
	\begin{multline*}
	\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2)+\left(r_M+2\sum_{j=1}^{M-1}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k)\\+2\sum_{j=1}^{M-1}\sum_{k=1}^{j-1}a_{j,k}+\sum_{k=1}^{M-1}r_{M}r_{k}\log \left( 1+ z_{k } \right).
	\end{multline*}
	Some little algebra then shows that
		\begin{align*}
		\frac{1}{2}\sum_{j=1}^{p} \sum_{k=1}^{p}  r_{j}r_{k}\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)=&2\sum_{j=1}^{M-1} \sum_{k=1}^{j-1}  r_{j}r_{k}\log (1 - z_{j}),\\
		&+2\sum_{j=1}^{M-1} \sum_{k=1}^{j-1}  r_{j}r_{k}\log (1+z_k),\\
		&+\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2),\\
		&+2\left(\sum_{j=1}^{M}r_j\right) \sum_{k=1}^{M-1}r_k\log(1+z_k).
		\end{align*}
		Changing the summation order and the name of the variables, the second term can be rewritten as
		\begin{multline*}
		2\sum_{j=1}^{M-2}r_j\log(1+z_j)\sum_{k=j+1}^{M-1}r_k=
		2\sum_{j=1}^{M-2}r_j\log(1+z_j)\left(\sum_{k=1}^{M-1}r_k-r_j-\sum_{k=1}^{j-1}r_k\right)\\=2\sum_{j=1}^{M-1}r_j\log(1+z_j)\left(\frac{N-r_M}{2}-1-r_j-\sum_{k=1}^{j-1}r_k\right)
		\end{multline*}
		We have then proved:
		\begin{align*}
		\frac{1}{2}\sum_{j=1}^{p} \sum_{k=1}^{p}  r_{j}r_{k}\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)=&2\sum_{j=1}^{M-1} r_{j}\log (1 - z_{j})\left(\sum_{k=1}^{j-1} r_{k} \right)\\
		&+2\sum_{j=1}^{M-1}r_j\log(1+z_j)\left(\frac{N-r_M}{2}-1-r_j-\sum_{k=1}^{j-1}r_k\right)\\
		&+\sum_{j=1}^{M-1}r_j^2\log(1-z_j^2)\\
		&+\left(N-2+r_M\right) \sum_{j=1}^{M-1}r_j\log(1+z_j).
		\end{align*}
		After simplification, we get
	\begin{align*}
	\frac{1}{2}\sum_{j=1}^{p} \sum_{k=1}^{p}  r_{j}r_{k}\log \left( 1 - z_{j}z_{k} + |z_{j} - z_{k }| \right)=&\sum_{j=1}^{M-1} r_{j}\left(r_j+2\sum_{k=1}^{j-1} r_{k} \right)\log(1 - z_{j})\\
	&-\sum_{j=1}^{M-1} r_{j}\left(r_j+2\sum_{k=1}^{j-1} r_{k} \right)\log (1+z_j)\\
	&+\left(2N-4\right) \sum_{j=1}^{M-1}r_j\log(1+z_j).
	\end{align*}
	Now we look at the first two terms recalling that
	\[
	z_j=1-\frac{1+r_j+2\sum_{k=1}^{j-1}r_k}{N-1}\Longrightarrow r_j+2\sum_{k=1}^{j-1}r_k=(N-1)(1-z_j)-1,
	\]
	and hence the sum in the lemma equals
	\begin{align*}
&(N-1)\sum_{j=1}^{M-1} r_{j}(1-z_j)\log(1 - z_{j})-\sum_{j=1}^{M-1} r_{j}\log(1 - z_{j})\\
&-(N-1)\sum_{j=1}^{M-1} r_{j}(1-z_j)\log (1+z_j)+\sum_{j=1}^{M-1} r_{j}\log(1 + z_{j})\\
&+\left(2N-4\right) \sum_{j=1}^{M-1}r_j\log(1+z_j),
	\end{align*}
	that is
	\begin{align*}
	&(N-1)\sum_{j=1}^{M-1} r_{j}(1-z_j)\log(1 - z_{j})-\sum_{j=1}^{M-1} r_{j}\log(1 - z_{j})\\
	&+(N-1)\sum_{j=1}^{M-1} r_{j}(1+z_j)\log (1+z_j)-\sum_{j=1}^{M-1} r_{j}\log(1 + z_{j}).\\
	\end{align*}
	The symmetry $z_j=z_{p+1-j}$ implies that the last expression equals
		\begin{align*}
		&(N-1)\sum_{j=1}^{M-1} r_{j}(1-z_j)\log(1 - z_{j})-\sum_{j=1}^{M-1} r_{j}\log(1 - z_{j})\\
		&+(N-1)\sum_{j=M+1}^{p} r_{j}(1-z_j)\log (1-z_j)-\sum_{j=M+1}^{p} r_{j}\log(1 - z_{j}).\\
		\end{align*}
		We thus have proved (using $z_M=0$) that the sum of the lemma equals
\[
(N-1)\sum_{j=1}^{p} r_{j}(1-z_j)\log(1 - z_{j})-\sum_{j=1}^{p} r_{j}\log(1 - z_j).
\]
\end{proof}

We now finally prove Theorem \ref{cor:nuevasuma}.
From Proposition \ref{PropEnergyOmegaGeneral} and Lemma \ref{lem3} we have
	\begin{equation*}
	\begin{split}
	&
	\Esp{\theta_{1},...,\theta_{p} \in \left[0,2\pi\right]^{p}}{\E{\log}(\Omega(p,r_{j}))}
	=
	-2\log(2)
	-(N-2)\log(4)
	-\frac12\sum_{j=1}^{p} r_j  \log(1 - z_{j}^{2})
	\\
	&
	-\sum_{j=1}^{p} r_{j}\log r_{j}
	-(N-1)\sum_{j=1}^{p} r_{j}(1-z_j)\log(1 - z_{j})+\sum_{j=1}^{p} r_{j}\log(1 - z_j)
	.
	\end{split}
	\end{equation*}
	Now, note that using $z_j=-z_{p+1-j}$ we have
	\begin{multline*}
	\frac12\sum_{j=1}^{p} r_j  \log(1 - z_{j}^{2})=\frac12\sum_{j=1}^{p} r_j  \log(1 - z_{j})+\frac12\sum_{j=1}^{p} r_j  \log(1 + z_{j})\\\frac12\sum_{j=1}^{p} r_j  \log(1 - z_{j})+\frac12\sum_{j=1}^{p} r_j  \log(1 - z_{j})=\sum_{j=1}^{p} r_j  \log(1 - z_{j}).
	\end{multline*}
	The theorem follows.

$\square$


text:
\section{Introduction}

Weight decay is an important trick and has been widely used in training Deep Neural Networks.
By constraining the weight magnitude, it is widely believed that weight decay regularizes the model and
improve generalization performance\citep{krizhevsky2012imagenet,bos1996optimal,bos1996using,krogh1992simple}.
Recently, a series of works \cite{van2017l2, zhang2018three, hoffer2018norm} propose that for layers followed by normalizations,
such as BatchNormalization\citep{ioffe2015batch},
the main effect of weight decay is increasing the \emph{effective learning rate}(ELR).
Take the widely adopted Conv-BN block as an example.
Since BatchNormalization is scale-invariant, the weight norm of the convolution layer do not actually affect the block's output,
which contradicts the regularization effect of weight decay.
On the contrary, the weight norm affects the step size of weight direction, e.g., larger weight norm results in smaller step size.
By constraining the weight norm from unlimitedly growing, weight decay is actually increasing the step size of weight direction,
thus increasing the ELR.

This interesting discovery arouses our questions: Does it covers the full mechanism(s) of weight decay?
If the answer is true, weight decay would be unnecessary since its effect fully overlaps with the original learning rate.
\citet{hoffer2018norm} shows that the performance can be recovered when training without weight decay by applying an LR correction technique.
However, the LR correction coefficient at each step depends on the original training statistics(with weight decay), which makes this technique only a verification of the effective learning rate hypothesis, but can not be used as a practical training method.
Moreover, the ELR hypothesis only applies to layers followed by normalizations, and there are layers that do not satisfy this requirement.
For example, the final fully-connected layer that commonly used in classification tasks.
For these layers, the effects of weight decay are usually omitted \citep{hoffer2018norm}, or simply the original weight decay is preserved \citep{zhang2018three}.
These problems indicate that the mechanisms of weight decay are still not fully understood.

In this paper we try to investigate the above problems.
For the convolution layers that followed by normalizations, we find that simply fixing the overall weight norm
to a constant fully recovers the effect of weight decay.
For the final fully-connected layer, we find that there is a special effect introduced by weight decay,
which influences the generalization performance by controlling the \textbf{cross-boundary risk}.
This mechanism is as important as the former investigated ELR, and they together capture most of the effects of weight decay.
These two mechanisms are unified into a new training scheme called \textbf{FixNorm},
which discards the weight decay and directly controls the effects of two main mechanisms.
By using FixNorm, we fully recover the performance of popular CNNs on large scale classification dataset ImageNet\citep{deng2009imagenet}.
Further, we show that the hyperparameters of FixNorm can be easily tuned, and propose a simple yet effective tuning method which
only requires a few trials and achieves near-optimal performance.
Specifically, by applying tuned FixNorm, we achieve 77.7\%(+0.4\%) with EfficientNet-B0, 79.5\%(+0.3\%) with EfficientNet-B1, 73.97\%(+1.9\%) with MobileNetV2.

Training tricks and network tricks show great impacts on performance, which also introduce difficulties in tuning and bring ambiguities to comparisons.
We show that this can be mitigated by using FixNorm.
For example, by simply scaling MobileNetV2 to the same FLOPS and applying the same tricks of EfficientNet-B0,
training with FixNorm achieves 77.4\% top-1 accuracy, while the default training process only get 76.72\%.
To facilitate fairer comparisons, we apply our FixNorm method to representative CNN architectures and set up new baselines under different settings.

Our contributions can be summarized as follows:
\begin{itemize}
  \item Except for increasing the effective learning rate, we discover a new mechanism of weight decay which controls the cross-boundary risk,
  and give a better understanding of weight decay's effect on generalization performance
  \item We propose a new training scheme called FixNorm that discards the weight decay and directly controls the effects of two main mechanisms,
  which not only fully recovers the accuracy of weight decay training, but also makes hyperparameters easier to tune.
  \item We propose a simple yet effective method to tune hyperparameters of FixNorm and demonstrate efficiency, robustness and SOTA performance on
  large scale datasets like ImageNet, MS COCO\citep{lin2014microsoft} and Cityscapes\citep{Cordts2016Cityscapes}
  \item By using our approach, we establish well-tuned baselines for popular networks, which we hope can facilitate fairer comparisons in the community
\end{itemize}

\section{Dissecting weight decay for training Deep Neural Networks} \label{sec_main}

\subsection{Revisiting the effective learning rate hypothesis} \label{sec_revisit}
We aim at further understanding the mechanisms of weight decay. Towards this, we first briefly revisit the effective learning rate(ELR) hypothesis.
As noted in \citet{hoffer2018norm}, when BN is applied after a linear layer, the output is invariant to the channel weight vector norm.
Denoting a channel weight vector with $\boldsymbol{w}$ and $\hat{\boldsymbol{w}}=\boldsymbol{w} / \norm{\boldsymbol{w}}$,
channel input as $\boldsymbol{x}$, we have
\begin{equation}
  \text{BN}(\norm{\boldsymbol{w}} \hat{\boldsymbol{w}} \boldsymbol{x}) = \text{BN}(\hat{\boldsymbol{w}} \boldsymbol{x})
\end{equation}
In such case, the gradient is scaled by $1 / \norm{\boldsymbol{w}}$:
\begin{equation}
  \frac{\partial \text{BN}(\norm{\boldsymbol{w}} \hat{\boldsymbol{w}} \boldsymbol{x})}{\partial(\norm{\boldsymbol{w}} \hat{\boldsymbol{w}})} = \frac{1}{\norm{\boldsymbol{w}}} \frac{\partial \text{BN}(\hat{\boldsymbol{w}} \boldsymbol{x})}{\partial \hat{\boldsymbol{w}}}
\end{equation}
This scale invariance makes the key feature of the weight vector is its \emph{direction}.
When the weights are updated through stochastic gradient descent at step $t$ and learning rate $\eta$
\begin{equation}
  \boldsymbol{w}_{t+1} = \boldsymbol{w}_{t} - \eta \nabla \boldsymbol{L}_t(\boldsymbol{w}_t)
\end{equation}
As in \cite{hoffer2018norm}, one can derive that the step size of the weight direction is approximately proportional to
\begin{equation}
  \boldsymbol{\hat{w}}_{t+1} - \boldsymbol{\hat{w}}_t \propto \frac{\eta}{\norm{\boldsymbol{w}_t}^2}
\end{equation}
Based on this formulation, the ELR hypothesis can be explained as follows: when applying weight decay to layers followed by normalization,
it prevents weight norm from unlimitedly growing, which preserves the step size of weight direction, thus ``increasing the effective learning rate''

However, this phenomenon is still not fully investigated.
\citet{hoffer2018norm} propose an LR correction technique that can train DNNs to similar performance without weight decay.
However, this LR correction technique needs to mimic the effective step size \emph{from training with weight decay}.
Similar techniques have also been proposed in \citet{zhang2018three}.
These techniques act as proofs of the ELR hypothesis, but can not be used as practical training methods.
On the other hand, as the hypothesis is based on the scale invariance brought by normalizations, there are layers that do not satisfy this precondition.
For example, the final fully-connected(FC) layers that commonly used in classification tasks.
As experiments in \citet{zhang2018three}(Figure 4), there is a clear gap between whether weight decay is applied to the FC layer.
These problems indicate that the mechanisms of weight decay are still not fully understood.

\subsection{Discarding weight decay for convolution layers} \label{sec_mechanism1}

We first consider discarding weight decay for convolution layers.
Since ELR hypothesis indicates that the main effect of weight decay on convolution layers is produced by constraining weight vector norm,
we investigate how weight vector norm changes during training.
Denoting the weights in all convolution layers as a single verctor $\bs{W}^{\text{Conv}}$, we plot $\norm{\bs{W}^{\text{Conv}}}$ in Fig \ref{fig:algo1}.
ResNet50\citet{he2016deep} is trained on ImageNet for 100 epochs with $lr=0.4$ and weght decay$\lambda=0.0001$. Other settings follow general setups in \ref{sec_exp}.

\begin{figure}[htbp]
  \centering
    \includegraphics[width=0.9\linewidth]{figures/fig1.pdf}
  \caption{\textbf{Left:} $\norm{\bs{W}^{\text{Conv}}}$ of WD training \textbf{Right:} top-1 accuracy for Algo \ref{algo1} and WD training,
  both learning rates are found by gridsearch}
  \label{fig:algo1}
\end{figure}

From Fig \ref{fig:algo1} left, the curve can be divided into two parts.
In the first part, $\norm{\bs{W}^{\text{Conv}}}$ decreases rapidly, which will increase the effective learning rate according to the ELR hypothesis.
However, we already adopt the learning rate warmup strategy, therefore this part of effect is duplicate and can be discarded.
In the second part, which occupies the majority of training, $\norm{\bs{W}^{\text{Conv}}}$ changes slowly in a relatively stable range.
From these observations, we propose to fix $\norm{\bs{W}^{\text{Conv}}}$ to a constant, as in Algo \ref{algo1}.
We rescale $\bs{W}^{\text{Conv}}$ after each optimization step,
which does not change outputs of the network but substantially maintains the effective learning rate.
While WD training controls the weight norm dynamically by a hyperparameter $\lambda$, Algo \ref{algo1} directly fixes the norm to a constant.
Since this constant can be any value, we choose it as the $\norm{\bs{W}^{\text{Conv}}}$ at initialization for simplicity.

\begin{algorithm}[h]
  \small
	\caption{Fixing the weight norm of convolution layers}
  \label{algo1}
  {\bf Input:} initial learning rate $lr$, total steps $T$, weight decay on final FC layer $\lambda_{\text{FC}}$, momentum $\mu$, training samples $\boldsymbol{x}$, corresponding labels $\boldsymbol{y}$ \\
	{\bf Initialization:} velocity $\boldsymbol{V}_{0} \leftarrow \boldsymbol{0}$, random initialize weight vector $\bs{W_{0}}$
\begin{algorithmic}[1]
  \For{$t$ in 0, ..., $T-1$}
    \State $\bs{x}, \bs{y}$ $\leftarrow$ BatchSampler($t$)
    \State $\widehat{\mathcal{L}}_{t}(\bs{W}_t) \leftarrow \sum \mathcal{L}(f(x;\bs{W}_t),y)+\frac{1}{2}\lambda_{\text{FC}} \norm{\bs{W}_{t}^{\text{FC}}}^2$
    \State $\bs{V}_{t+1} \leftarrow \mu \bs{V}_t+ \nabla \widehat{\mathcal{L}}_{t}(\bs{W}_t)$

    \State $\eta_{t} \leftarrow $ GetLRScheduleMultiplier$(t)$
    \State $\bs{W}_{t+1} \leftarrow \bs{W}_{t} -lr \times \eta_t \times \bs{V}_{t+1}$

    \State $\bs{W}^{\text{Conv}}_{t+1} \leftarrow \frac{ \bs{W}^{\text{Conv}}_{t+1} }{ \norm{\bs{W}^{\text{Conv}}_{t+1}} } \norm{\bs{W}^{\text{Conv}}_{0}}$
  \EndFor
\end{algorithmic}
\end{algorithm}

Since the fixed weight norm in Algo \ref{algo1} is substantially different from WD training, their optimal learning rates are different as well.
To be fair, we grid search the $lr$ for both Algo \ref{algo1} and WD training and compare the best performance.
As shown in Fig \ref{fig:algo1} right, Algo \ref{algo1} achieves same top-1 accuracy of WD training.
These results demonstrate that for convolution layers followed by BN, weight decay can be discarded.
Note that we preserve $\lambda^{\text{FC}}$ for the final FC layer, which will be addressed in the next subsection.

\vspace{-3mm}

\subsection{Effects on final fully-connected layers} \label{sec_mechanism2}

As noted in section \ref{sec_revisit}, the final FC layer does not satisfy the precondition of the ELR hypothesis.
To investigate the effects of weight decay on this layer, we first try to make it scale-invariant.
We apply three modifications on Algo \ref{algo1}: (1) replace original FC layer with WN-FC layer;
(2) replace $\bs{W}^{\text{Conv}}$ in line 7 of Algo \ref{algo1} with $\bs{W}^{\text{Conv+FC}}$; (3) set $\lambda^{\text{FC}}=0$.
This modified algorithm is denoted as Algo 1@WN-FC.
\footnote{For full algorithm descriptions please refer to Algo 2 in supplementary materials}
WN-FC layer is normal FC layer applied with weight normalization\citep{salimans2016weight}. Original FC and WN-FC layer are formulated as follows:

\vspace{-2mm}

\begin{align}
  \text{FC}(x; \bs{W}^{\text{FC}}) &= \trans{x} \bs{W}^{\text{FC}} \\
  \text{WN-FC}(x; \bs{W}^{\text{FC}}, g) &= \frac{\trans{x} \bs{W}^{\text{FC}}}{\norm{\bs{W}^{\text{FC}}}} \times g \label{eqa:wn-fc}
\end{align}

\begin{figure}[htbp]
  \centering
    \includegraphics[width=0.9\linewidth]{figures/fig2.pdf}
  \caption{Training ResNet50 on ImageNet with Algo \ref{algo1}, Algo 1@WN-FC and Algo 1@FixNorm-FC.
  \textbf{Left:} top-1 accuracy \textbf{Top right:} weight norm of the final FC layer ($\norm{\bs{W}^\text{FC}}$ for FC, $g$ for WN-FC and FixNorm-FC)
  \textbf{Bottom right:} MCBR}
  \label{fig:algo123}
\end{figure}

We compare Algo \ref{algo1} and Algo 1@WN-FC by training ResNet50 on ImageNet.
As can be seen in Fig \ref{fig:algo123} left, there is a clear gap between Algo \ref{algo1} and Algo 1@WN-FC,
which implies that weight decay has additional effects beyond preserving ELR on final FC layers.

Now lets combine equation \ref{eqa:wn-fc} with softmax cross-entropy loss $\mathcal{L}$,
$s_i= \frac{\trans{x} \bs{W}_i}{\norm{\bs{W}}}g$ denotes the logits value of class $i$,
$p_i$ denotes the probability of class $i$, $k$ for the label class and $j$ for other classes.
We have(for full derivations please refer to supplymentary materials),
\begin{align}
  \mathcal{L}(x, k) &= - \log p_k = - \log \frac{e^{s_k}}{\sum e^{s_j}} \\
  -\frac{\partial \mathcal{L}(x, k)}{\partial x} &= \frac{g}{\norm{\bs{W}}} \sum_{j \neq k} p_j(\bs{W}_k - \bs{W}_j)
\end{align}

\begin{wrapfigure}[11]{r}{0.40\textwidth}
  \centering
    \includegraphics[width=0.95\linewidth]{figures/fig3.pdf}
  \caption{Ilustration of feature space}
  \label{fig:boundary}
\end{wrapfigure}

Given $\bs{W}$, the gradient is actually driving $x$ from other class center $\bs{W}_j$ towards label class center $\bs{W}_k$,
where the magnitude depends on $p_j$ and $g$.
Note that $p_j$ also depends on $g$ through the softmax function.
When $x$ is correctly classified and $g$ continuously grows, $p_j$ will rapidly shrink and weaken the gradient.
As illustrated in Fig \ref{fig:boundary}, this will leave $x$ being closer to the class boundary between
$\bs{W}_j$ and $\bs{W}_k$(larger $\cos \beta$), which is \emph{less discriminative}.
This ambiguous feature space is prone to distribution shift between training and testing, therefore may result in poor generalization.

To quantitively verify this explaination, we define \textbf{Mean Cross-Boundary Risk}(MCBR):
\begin{equation}
  \text{MCBR}(x, k, \bs{W}) = \frac{1}{\#class-1} \sum_{j\neq k} \cos(x, \bs{W}_j-\bs{W}_k)
\end{equation}
MCBR shows how much $x$ is lean to the class boundaries, ranging from -1 to 1.
The larger MCBR is, the more likely $x$ will cross the class boundaries during testing.
We compare the weight norm of FC layer($g$ for WN-FC layer) and MCBR for Algo \ref{algo1} and Algo 1@WN-FC in Fig \ref{fig:algo123} top right and bottom right.
It can be clearly observed that without constraint, $g$ continuously grows and leads to higher MCBR.
This explains why Algo 1@WN-FC generalize poorly.

Based on these analysis, we propose to constrain $g$ from exceeding a given upperbound $\alpha$, denoted as FixNorm-FC.
$\sqrt{\#class}$ normalize the upper bound across different number of classes.
\begin{equation}
  \text{FixNorm-FC}(x; \bs{W}^{FC}, g, \alpha) = \frac{\trans{x} \bs{W}^{FC}}{\norm{\bs{W}^{FC}}} \times \min (g, \alpha * \sqrt{\#class}) \label{eqa:fixnorm-fc}
\end{equation}
Note that $\bs{W}^{FC}$ and $g$ are learnable parameters, while $\alpha$ is a hyper-parameter.
We replace the WN-FC layer in Algo 1@WN-FC with the FixNorm-FC layer, denoted as Algo 1@FixNorm-FC.\footnote{For full algorithm descriptions please refer to Algo 1@FixNorm-FC in supplementary materials}
We choose $\alpha$ according to the weight norm of Algo \ref{algo1} in Fig \ref{fig:algo123} top right and leave other hyper-parameters unchanged, the results are shown in Figure \ref{fig:algo123} left.
By simply constraining the upper limit of $g$, Algo 1@FixNorm-FC maintains low MCBR and fully closes the accuracy gap.

Moreover, we find that the optimal values for $\alpha$ are different among models.
We give more comprehensive experiments in Section \ref{sec_exp}.
To summarise, for the final FC layer, the ELR hypothesis does not cover all the effects of weight decay.
We find that weight decay influences the \textbf{cross-boundary risk} by constraining the FC layer's weight norm and finally affects generalization performance.
By using the FixNorm-FC layer, Algo 1@FixNorm-FC can fully recover the accuracy of normal weight decay training.
Moreover, Algo 1@FixNorm-FC directly controls the two main mechanisms, which makes the hyperparameters more easier to tune.
We will show this in section \ref{sec_tuning} and \ref{sec_exp_tune}.

\subsection{Tuning lr and alpha for FixNorm training} \label{sec_tuning}

\begin{wrapfigure}[14]{r}{0.40\textwidth}
  \centering
    \includegraphics[width=0.95\linewidth]{figures/fig4.pdf}
  \caption{Top-1 accuracy for different $lr$ and $\alpha$. ResNet50 trained for 50 epochs}
  \label{fig:corr}
\end{wrapfigure}

Section \ref{sec_mechanism1} and \ref{sec_mechanism2} investigate the two main mechanisms of weight decay:
(1) for layers followed by normalizations(mainly convolution layers), affecting ELR (2) for final fully-connected layers, affecting cross-boundary risk.
Algo 1@FixNorm-FC (refered to \textbf{FixNorm} for simplicity) unifies these two mechanisms and directly controls
their effects through hyper-parameters $lr$ and $\alpha$.
While these two mechanisms both affect generalization performance, it is important to know how they are correlated,
which will determine how to tune two hyper-parameters.
To verify this, we grid search $lr$ and $\alpha$ and show the corresponding top-1 accuracy in Fig \ref{fig:corr}.
It clearly shows that the best $lr$ does not depends on the value of $\alpha$ and vice versa.
This suggests that we can tune $lr$ and $\alpha$ \emph{independently}, which will greatly reduce the cost.

Instead of using existing hyper-parameter optimization(HPO) methods, we propose a simple yet effective approach to tune $lr$ and $\alpha$.
We introduce two priors to efficiently tune $lr$.
\begin{itemize}
  \item Top-1 accuracy is approximately a convex function for $lr$
  \item The best $lr$ for shorter training is usually larger than that for longer training
\end{itemize}
The first prior is mainly an empirical finding, while the second one may be partially explained by
the correlation between generalization performance and weight distance from their initialization\citep{trainloger2017nips}:
shorter training may require larger $lr$ to travel far enough from the initialization in weight space to generalize well.
These two priors motivate us to \emph{use best lr under shorter training as an upper bound for that under longer training}.
This strategy can adaptively shrink the search range and let us locate the best $lr$ in a wide range with relatively low cost.
After the best $lr$ is found, we simply fix it and grid search for the best $\alpha$.
The overall method can be summarized in Algo \ref{algo4}(\textbf{tuned FixNorm}).

\setcounter{algorithm}{3}

\begin{algorithm}[h]
  \small
	\caption{Tuning $lr$ and $\alpha$ for FixNorm training}
  \label{algo4}
  {\bf Input:} number of $lr$ tuning rounds $N$, learning rate range $[lr_{min}, lr_{max}]$, learning rate split number $K$, training steps of each $lr$ tuning round $\bs{T}=[T_0, T_1, ..., T_{N-1}]$ where $T_i \leq T_{i+1} $, alpha candidates $[\alpha_0, \alpha_1, ..., \alpha_{m-1}]$ \\
  {\bf Output:} $\alpha_{best}$, $lr_{best}$, $acc_{best}$ \\
  {\bf Initialization:} $\alpha_{best} = \alpha_0$, $lr_{best} = \text{NULL}$, $acc_{best} = 0$
\begin{algorithmic}[1]
  \Algphase{Phase 1 - Find $lr_{best}$}
  \For{$r$ in 0, ..., $N-1$}
    \State $\bs{LR} \leftarrow \text{UniformSplit}(lr_{min}, lr_{max}, K)$
    \State $\bs{Acc} \leftarrow \left\{ \text{FixNormTrain}(LR_k, \alpha_{best}, T_r)|\ k \in\left\{0, ..., K-1\right\} \right\}$
    \State $idx \leftarrow \arg \max \bs{Acc}$
    \State $lr_{max} \leftarrow \bs{LR}_{idx}$
    \If{$\bs{Acc}_{idx} > acc_{best} $}
      \State $acc_{best} \leftarrow \bs{Acc}_{idx}$
      \State $lr_{best} \leftarrow \bs{LR}_{idx}$
    \EndIf
  \EndFor
  \Algphase{Phase 2 - Find $\alpha_{best}$}
  \State $\bs{Acc} \leftarrow \left\{ \text{FixNormTrain}(lr_{best}, \alpha_{i}, T_{N-1})|\ i \in \left\{1, ..., m-1\right\} \right\}$
  \State $idx \leftarrow \arg \max \bs{Acc}$
  \If{$\bs{Acc}_{idx} > acc_{best} $}
    \State $acc_{best} \leftarrow \bs{Acc}_{idx}$
    \State $\alpha_{best} \leftarrow \alpha_{idx}$
  \EndIf
\end{algorithmic}
\end{algorithm}

\vspace{-1mm}

\section{Experiments} \label{sec_exp}

\paragraph{General setups}
We perform experiments on ImageNet classification task \citep{deng2009imagenet} which contains 1.28 million training images and 50000 validation images.
Our general training settings are mainly adapted from \citet{he2019bag}, which include Nesterov Accelerated Gradient (NAG) descent\citep{nesterov1983method},
one-cycle cosine learning rate decay\citep{loshchilov2016sgdr} with linear warmup at first 4 epochs\citep{goyal2017accurate} and
label smoothing with $\epsilon=0.1$\citep{szegedy2016rethinking}.
We do not use mixup augmentation\citep{zhang2017mixup}.
All the models are trained on 16 Nvidia V100 GPUs with a total batch size of 1024. Other settings follow reference implementations of each model.
We leave experiments on MS COCO and Cityscapes in supplementary materials.

\vspace{-1mm}

\paragraph{FixNorm tuning setups}
For Algo \ref{algo4}, we set $N=2$, learning rate range $[0.2, 3.2]$, $K=5$, $\bs{T}=[0.2T_{max}, T_{max}]$ where $T_{max}$ is the max training steps, $\alpha$ candidates [0.5, 1.0, 2.0, 4.0, 8.0, 16.0].
The search contains two $lr$ tuning rounds and one $\alpha$ tuning round.
The total computational resources consumed are $K \times 0.2T_{max} + K \times T_{max} + (6-1) \times T_{max} = 11T_{max}$, which is 11 times of a single training process.

\vspace{-1mm}

\subsection{Tuned FixNorm on ResNet50-D and MobileNetV2} \label{sec_exp_tune}
To demonstrate the effectiveness of our method, we first apply Algo \ref{algo4} on two well-studied architectures: ResNet50-D\citep{he2019bag} and MobileNetV2\citep{sandler2018mobilenetv2}.
We follow \citet{he2019bag} and train 120 epochs for ResNet50-D and 150 epochs for MobileNetV2.
Their reference top-1 accuracies reported are 78.37\% and 72.04\%.
We also adopt Bayesian Optimization(BO)\citep{bo_nips2012} to search learning rate and weight decay under normal weight decay training (BO+WD) for these models.
We use the same learning rate range $[0.2, 3.2]$ for BO, and the weight decay range is set to $[0.00001, 0.0005]$.
Results are shown in Table \ref{tb:exp1}.

\vspace{-2mm}

\begin{table}[h]
  \centering
  \caption{Results for tuned FixNorm and BO+WD training}
  \label{tb:exp1}
  \resizebox{0.85\textwidth}{!}{

    \begin{tabular}{cccccccccc}
      \toprule
                    & \multicolumn{3}{c}{Tuned FixNorm} & \multicolumn{3}{c}{BO + WD} & \multicolumn{3}{c}{Reference}\\
      \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
                  & $lr$   & $\alpha$ & top-1(\%)   & $lr$   & $\lambda$ & top-1(\%) & $lr$   & $\lambda$ & top-1(\%) \\
      \midrule
      ResNet50-D   & 1.4    & 0.5 & \textbf{78.62}  & 0.53   &  8.6e-5   & 78.53     & 0.4    & 1e-4      & 78.37     \\
      MobileNetV2  & 0.5    & 16.0 & \textbf{73.20} & 0.64   &  2.2e-5   & 72.84     & 0.2    & 4e-5      & 72.04     \\
      \bottomrule
    \end{tabular}

  }
\end{table}

\vspace{-3mm}

As in Table \ref{tb:exp1}, both tuned FixNorm and BO+WD outperform reference settings by a clear margin.
Although the reference settings have already been heavily refined in \citet{he2019bag}, our method still brings substantial improvements.
We further compare tuned FixNorm and BO+WD in Fig \ref{fig:exp1} left.
Our method shows two advantages compared to BO+WD.
First, our method finds better solutions at lower cost.
The cost of our method is 11 times of normal training while BO+WD requires 25 and 35, while our final results are even better than that found by BO+WD.
Second, our method is more stable and barely needs meta-tuning.
BO itself has lots of tunable meta hyper-parameters \citep{lindauer2019towards} and it requires expert knowledge to tune them.
While we use exactly the same FixNorm tuning setups for all the experiments, including in Table \ref{tb:exp2}.
These setups are intuitive and the method performs consistently well across all the settings.

\vspace{-2mm}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\linewidth]{figures/fig5.pdf}
  \caption{
    \textbf{Left:} search progress for FixNorm and BO+WD
    \textbf{Middle:} FixNorm search details for ResNet50-D
    \textbf{Right:} FixNorm search details for MobileNetV2
  }
  \label{fig:exp1}
\end{figure}
\vspace{-1mm}

To better understand our method, we show more details about tuned FixNorm in Fig \ref{fig:exp1} middle and right, for ResNet50-D and MobileNetV2 respectively.
There are two $lr$ searching rounds and one $\alpha$ searching round under our setups.
In the first $lr$ round, both experiments starts with $lr$ values in [0.8, 1.4, 2.0, 2.6, 3.2] (which are the uniform split points of initial range [0.2, 3.2])
and train for $1/5$ total epochs(24 epochs and 30 epochs respectively).
Best $lr$ values are very different in this round: $lr_1=2.6$ for ResNet50-D and $lr_1=0.8$ for MobileNetV2.
Taking these values as new upper bound, [0.2, $lr_1$] are further split and corresponding $lr$ values are evaluated in the second round, for full total epochs.
The best $lr$ values are 1.4 and 0.5 in this round.
These values are fixed and then $\alpha$ is searched.
For ResNet50-D, the initial $\alpha=0.5$ is already the best, while for MobileNetV2 a better $\alpha=16.0$ is founded.
From Fig \ref{fig:exp1} one can find the patterns match two priors introduced in section \ref{sec_tuning}.

\vspace{-2mm}

\subsection{New state-of-the-arts with tuned FixNorm}

Many powerful networks have been proposed recently.
These networks usually adopt many tricks and therefore hard to tune.
To fully exploit the capabilities of these networks, we apply tuned FixNorm to further optimize them.
We also apply advanced tricks to basic models like MobileNetV2 and ResNet50-D.
These tricks are used by EfficientNet\citep{tan2019efficientnet}, including SE-layer\citep{hu2018squeeze}, swish activation\citep{ramachandran2017searching},
stochastic depth training\citep{huang2016deep} and AutoAugment preprocessing\citep{cubuk2019autoaugment}.
From Table \ref{tb:exp2}, we can find that:
\begin{itemize}
  \item Our method consistently outperforms reference settings. Strong baselines like EfficientNet can be further improved by our method, specifically +0.4\% and +0.3\% for B0 and B1.
  \item Tunning matters. When simply apply tricks to MobileNetV2 and scale to the same FLOPS with B0 and B1, tuned FixNorm achieves 77.4\% and 79.18\% top-1 accuracy,
  while default training settings only get 76.72 and 78.75. This difference can lead to unreliable conclusions when compared to EfficientNet.
  \item Best $lr$ and $\alpha$ are different among models, even for the same model with different settings. This may suggest that we should tune for each setting to fully exploit their performance.
\end{itemize}

\vspace{-2mm}

\begin{table}[h]
  \centering
  \captionof{table}{
    \textbf{Results with tuned FixNorm} models with * are applied with tricks,
    \dag means the result is obtained using $lr$ and $\lambda$ from basic settings(MobileNetV2 150 and ResNet50-D 120)
  }
  \label{tb:exp2}
  \resizebox{0.95\textwidth}{!}{

    \begin{tabular}{ccccrrcc}
      \toprule
      Model                    &\#Epochs & Top-1      & Top-1 (ref.)  & \#Params    & \#FLOPS  & $lr$  & $\alpha$ \\
      \midrule
      MobileNetV2              & 150     & 73.20      & 72.04         &  3.5M       & 300M     & 0.5   & 16.0     \\
      MobileNetV2              & 350     & 73.97      & 73.38$^\dag$  &  3.5M       & 300M     & 0.35  & 8.0      \\
      EfficientNet-B0          & 350     & 77.72      & 77.30         &  5.3M       & 384M     & 0.5   & 4.0      \\
      EfficientNet-B1          & 350     & 79.52      & 79.20         &  7.8M       & 685M     & 0.8   & 8.0      \\
      MobileNetV2$\times$1.12* & 350     & 77.40      & 76.72$^\dag$  &  4.7M       & 386M     & 0.5   & 8.0     \\
      MobileNetV2$\times$1.54* & 350     & 79.18      & 78.75$^\dag$  &  8.0M       & 682M     & 0.65  & 4.0     \\
      ResNet50-D               & 120     & 78.62      & 78.37         &  25.6M      & 4.3G     & 1.4   & 0.5     \\
      ResNet50-D               & 350     & 79.29      & 79.04$^\dag$  &  25.6M      & 4.3G     & 1.1   & 0.5     \\
      ResNet50-D*              & 350     & 81.27      & 80.80$^\dag$  &  28.1M      & 4.3G     & 1.1   & 1.0     \\
      \bottomrule
    \end{tabular}
  }

\end{table}

\vspace{-2mm}

\section{Related works}

Due to the page limit, we only highlight the most related works in this section and leave other works in supplementary materials.

\paragraph{Understanding weight decay} Recently, a series of works \citep{van2017l2, zhang2018three, hoffer2018norm} propose that when combined with normalizations,
the main effect of weight decay is increasing ELR, which is contrary to the previous understanding and motivates new perspectives.
\citet{van2017l2} first introduces the ELR hypothesis and provides derivations for different optimizers,
while both \citet{hoffer2018norm,zhang2018three} give additional evidence supporting the hypothesis.
\citet{hoffer2018norm} also proposes norm-bounded Weight Normalization, which fixes the norm of each convolution layer separately.
By doing this, their method fixes the ELR of each layer, which \emph{highly depends on the initialization of each layer}.
Differently, we fix the norm of all convolution layers as a whole and maintains the \emph{global} ELR,
which is more robust and demonstrates SOTA performance on large scale experiments.
Layer-wise ELR controlling is an interesting problem and may lead to new perspectives for weight initialization techniques.
Similar to \citet{hoffer2018norm}, \citet{xiang2019understanding} also proposes modifications to Weight Normalization based on ELR hypothesis.
They identify the problems when using weight decay with Weight Normalization, and propose $\epsilon-$shifted $L_2$ regularizer
to constrain weight norm to $\epsilon$ with coefficient $\lambda$.
Beyond ELR hypothesis, \citet{li2019exponential} derives a closed-form between learning rate, weight decay and momentum,
and proposes an exponentially increasing learning rate schedule.
Their work mainly discusses the linkage of three hyper-parameters, while our work focuses on the underlying mechanisms of weight decay.
Except for the ELR hypothesis, \citet{loshchilov2017decoupled} identifies problems when applying weight decay to Adam optimizer,
which improves generalization performance and decouples it from learning rate.
All these works bring interesting perspectives for understanding weight decay, yet our work has distinct differences and contributions.
First, our work investigates the effect on final FC layers and find a new mechanism that complements the understanding of weight decay
on generalization performance, which is mostly ignored by previous works.
Second, our method including FixNorm and the tuning method are both concise and effective, and demonstrate SOTA performance on large scale datasets.

\section{Conclusion}
In this paper, we find a new mechanism of weight decay on final FC layers,
which affects generalization performance by controlling \emph{cross-boundary risk}.
This new mechanism complements the ELR hypothesis and gives a better understanding of weight decay.
We propose a new training method called \textbf{FixNorm}, which discards weight decay and directly controls the two mechanisms.
We also propose an effective, efficient and robust method to tune hyperparameters of FixNorm, which can consistently find near-optimal solutions in a few trials.
Experiments on large scale datasets demonstrate our methods, and a series of SOTA baselines are established for fair comparisons.
We believe this work brings new perspectives and may motivate interesting ideas like controlling layer-wise ELR and automatically adjusting cross-boundary risk.

\section{Submission of conference papers to ICLR 2021}

ICLR requires electronic submissions, processed by
\url{https://openreview.net/}. See ICLR's website for more instructions.

If your paper is ultimately accepted, the statement {\tt
  {\textbackslash}iclrfinalcopy} should be inserted to adjust the
format to the camera ready requirements.

The format for the submissions is a variant of the NeurIPS format.
Please read carefully the instructions below, and follow them
faithfully.

\subsection{Style}

Papers to be submitted to ICLR 2021 must be prepared according to the
instructions presented here.

Authors are required to use the ICLR \LaTeX{} style files obtainable at the
ICLR website. Please make sure you use the current files and
not previous versions. Tweaking the style files may be grounds for rejection.

\subsection{Retrieval of style files}

The style files for ICLR and other conference information are available online at:
\begin{center}
   \url{http://www.iclr.cc/}
\end{center}
The file \verb+iclr2021_conference.pdf+ contains these
instructions and illustrates the
various formatting requirements your ICLR paper must satisfy.
Submissions must be made using \LaTeX{} and the style files
\verb+iclr2021_conference.sty+ and \verb+iclr2021_conference.bst+ (to be used with \LaTeX{}2e). The file
\verb+iclr2021_conference.tex+ may be used as a ``shell'' for writing your paper. All you
have to do is replace the author, title, abstract, and text of the paper with
your own.

The formatting instructions contained in these style files are summarized in
sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.

\section{General formatting instructions}
\label{gen_inst}

The text must be confined within a rectangle 5.5~inches (33~picas) wide and
9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).
Use 10~point type with a vertical spacing of 11~points. Times New Roman is the
preferred typeface throughout. Paragraphs are separated by 1/2~line space,
with no indentation.

Paper title is 17~point, in small caps and left-aligned.
All pages should start at 1~inch (6~picas) from the top of the page.

Authors' names are
set in boldface, and each name is placed above its corresponding
address. The lead author's name is to be listed first, and
the co-authors' names are set to follow. Authors sharing the
same address can be on the same line.

Please pay special attention to the instructions in section \ref{others}
regarding figures, tables, acknowledgments, and references.

There will be a strict upper limit of 8 pages for the main text of the initial submission, with unlimited additional pages for citations. Note that the upper page limit differs from last year!Authors may use as many pages of appendices (after the bibliography) as they wish, but reviewers are not required to read these. During the rebuttal phase and for the camera ready version, authors are allowed one additional page for the main text, for a strict upper limit of 9 pages.

\section{Headings: first level}
\label{headings}

First level headings are in small caps,
flush left and in point size 12. One line space before the first level
heading and 1/2~line space after the first level heading.

\subsection{Headings: second level}

Second level headings are in small caps,
flush left and in point size 10. One line space before the second level
heading and 1/2~line space after the second level heading.

\subsubsection{Headings: third level}

Third level headings are in small caps,
flush left and in point size 10. One line space before the third level
heading and 1/2~line space after the third level heading.

\section{Citations, figures, tables, references}
\label{others}

These instructions apply to everyone, regardless of the formatter being used.

\subsection{Citations within the text}

Citations within the text should be based on the \texttt{natbib} package
and include the authors' last names and year (with the ``et~al.'' construct
for more than two authors). When the authors or the publication are
included in the sentence, the citation should not be in parenthesis using \verb|\citet{}| (as
in ``See \citet{Hinton06} for more information.''). Otherwise, the citation
should be in parenthesis using \verb|\citep{}| (as in ``Deep learning shows promise to make progress
towards AI~\citep{Bengio+chapter2007}.'').

The corresponding references are to be listed in alphabetical order of
authors, in the \textsc{References} section. As to the format of the
references themselves, any style is acceptable as long as it is used
consistently.

\subsection{Footnotes}

Indicate footnotes with a number\footnote{Sample of the first footnote} in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2~inches
(12~picas).\footnote{Sample of the second footnote}

\subsection{Figures}

All artwork must be neat, clean, and legible. Lines should be dark
enough for purposes of reproduction; art work should not be
hand-drawn. The figure number and caption always appear after the
figure. Place one line space before the figure caption, and one line
space after the figure. The figure caption is lower case (except for
first word and proper nouns); figures are numbered consecutively.

Make sure the figure caption does not get separated from the figure.
Leave sufficient space to avoid splitting the figure and figure caption.

You may use color figures.
However, it is best for the
figure captions and the paper body to make sense if the paper is printed
either in black/white or in color.
\begin{figure}[h]
\begin{center}
\fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
\end{center}
\caption{Sample figure caption.}
\end{figure}

\subsection{Tables}

All tables must be centered, neat, clean and legible. Do not use hand-drawn
tables. The table number and title always appear before the table. See
Table~\ref{sample-table}.

Place one line space before the table title, one line space after the table
title, and one line space after the table. The table title must be lower case
(except for first word and proper nouns); tables are numbered consecutively.

\begin{table}[t]
\caption{Sample table title}
\label{sample-table}
\begin{center}
\begin{tabular}{ll}
\multicolumn{1}{c}{\bf PART}  &\multicolumn{1}{c}{\bf DESCRIPTION}
\\ \hline \\
Dendrite         &Input terminal \\
Axon             &Output terminal \\
Soma             &Cell body (contains cell nucleus) \\
\end{tabular}
\end{center}
\end{table}

\section{Default Notation}

In an attempt to encourage standardized notation, we have included the
notation file from the textbook, \textit{Deep Learning}
\cite{goodfellow2016deep} available at
\url{https://github.com/goodfeli/dlbook_notation/}.  Use of this style
is not required and can be disabled by commenting out
\texttt{math\_commands.tex}.

\centerline{\bf Numbers and Arrays}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1in}p{3.25in}}
$\displaystyle a$ & A scalar (integer or real)\\
$\displaystyle {\bm{a}}$ & A vector\\
$\displaystyle {\bm{A}}$ & A matrix\\
$\displaystyle {\tens{A}}$ & A tensor\\
$\displaystyle {\bm{I}}_n$ & Identity matrix with $n$ rows and $n$ columns\\
$\displaystyle {\bm{I}}$ & Identity matrix with dimensionality implied by context\\
$\displaystyle {\bm{e}}^{(i)}$ & Standard basis vector $[0,\dots,0,1,0,\dots,0]$ with a 1 at position $i$\\
$\displaystyle \text{diag}({\bm{a}})$ & A square, diagonal matrix with diagonal entries given by ${\bm{a}}$\\
$\displaystyle {\textnormal{a}}$ & A scalar random variable\\
$\displaystyle {\mathbf{a}}$ & A vector-valued random variable\\
$\displaystyle {\mathbf{A}}$ & A matrix-valued random variable\\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Sets and Graphs}
\bgroup
\def1.5{1.5}

\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle {\mathbb{A}}$ & A set\\
$\displaystyle \mathbb{R}$ & The set of real numbers \\
$\displaystyle \{0, 1\}$ & The set containing 0 and 1 \\
$\displaystyle \{0, 1, \dots, n \}$ & The set of all integers between $0$ and $n$\\
$\displaystyle [a, b]$ & The real interval including $a$ and $b$\\
$\displaystyle (a, b]$ & The real interval excluding $a$ but including $b$\\
$\displaystyle {\mathbb{A}} \backslash {\mathbb{B}}$ & Set subtraction, i.e., the set containing the elements of ${\mathbb{A}}$ that are not in ${\mathbb{B}}$\\
$\displaystyle {\mathcal{G}}$ & A graph\\
$\displaystyle \parents_{\mathcal{G}}({\textnormal{x}}_i)$ & The parents of ${\textnormal{x}}_i$ in ${\mathcal{G}}$
\end{tabular}
\vspace{0.25cm}

\centerline{\bf Indexing}
\bgroup
\def1.5{1.5}

\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle {a}_i$ & Element $i$ of vector ${\bm{a}}$, with indexing starting at 1 \\
$\displaystyle {a}_{-i}$ & All elements of vector ${\bm{a}}$ except for element $i$ \\
$\displaystyle {A}_{i,j}$ & Element $i, j$ of matrix ${\bm{A}}$ \\
$\displaystyle {\bm{A}}_{i, :}$ & Row $i$ of matrix ${\bm{A}}$ \\
$\displaystyle {\bm{A}}_{:, i}$ & Column $i$ of matrix ${\bm{A}}$ \\
$\displaystyle {\etens{A}}_{i, j, k}$ & Element $(i, j, k)$ of a 3-D tensor ${\tens{A}}$\\
$\displaystyle {\tens{A}}_{:, :, i}$ & 2-D slice of a 3-D tensor\\
$\displaystyle {\textnormal{a}}_i$ & Element $i$ of the random vector ${\mathbf{a}}$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Calculus}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle\frac{d y} {d x}$ & Derivative of $y$ with respect to $x$\\ [2ex]
$\displaystyle \frac{\partial y} {\partial x} $ & Partial derivative of $y$ with respect to $x$ \\
$\displaystyle \nabla_{\bm{x}} y $ & Gradient of $y$ with respect to ${\bm{x}}$ \\
$\displaystyle \nabla_{\bm{X}} y $ & Matrix derivatives of $y$ with respect to ${\bm{X}}$ \\
$\displaystyle \nabla_{\tens{X}} y $ & Tensor containing derivatives of $y$ with respect to ${\tens{X}}$ \\
$\displaystyle \frac{\partial f}{\partial {\bm{x}}} $ & Jacobian matrix ${\bm{J}} \in \mathbb{R}^{m\times n}$ of $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$\\
$\displaystyle \nabla_{\bm{x}}^2 f({\bm{x}})\text{ or }{\bm{H}}( f)({\bm{x}})$ & The Hessian matrix of $f$ at input point ${\bm{x}}$\\
$\displaystyle \int f({\bm{x}}) d{\bm{x}} $ & Definite integral over the entire domain of ${\bm{x}}$ \\
$\displaystyle \int_{\mathbb{S}} f({\bm{x}}) d{\bm{x}}$ & Definite integral with respect to ${\bm{x}}$ over the set ${\mathbb{S}}$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Probability and Information Theory}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle P({\textnormal{a}})$ & A probability distribution over a discrete variable\\
$\displaystyle p({\textnormal{a}})$ & A probability distribution over a continuous variable, or over
a variable whose type has not been specified\\
$\displaystyle {\textnormal{a}} \sim P$ & Random variable ${\textnormal{a}}$ has distribution $P$\\% so thing on left of \sim should always be a random variable, with name beginning with \r
$\displaystyle  \mathbb{E}_{{\textnormal{x}}\sim P} [ f(x) ]\text{ or } \mathbb{E} f(x)$ & Expectation of $f(x)$ with respect to $P({\textnormal{x}})$ \\
$\displaystyle \mathrm{Var}(f(x)) $ &  Variance of $f(x)$ under $P({\textnormal{x}})$ \\
$\displaystyle \mathrm{Cov}(f(x),g(x)) $ & Covariance of $f(x)$ and $g(x)$ under $P({\textnormal{x}})$\\
$\displaystyle H({\textnormal{x}}) $ & Shannon entropy of the random variable ${\textnormal{x}}$\\
$\displaystyle D_{\mathrm{KL}} ( P \Vert Q ) $ & Kullback-Leibler divergence of P and Q \\
$\displaystyle \mathcal{N} ( {\bm{x}} ; {\bm{\mu}} , {\bm{\Sigma}})$ & Gaussian distribution %
over ${\bm{x}}$ with mean ${\bm{\mu}}$ and covariance ${\bm{\Sigma}}$ \\
\end{tabular}
\egroup
\vspace{0.25cm}

\centerline{\bf Functions}
\bgroup
\def1.5{1.5}
\begin{tabular}{p{1.25in}p{3.25in}}
$\displaystyle f: {\mathbb{A}} \rightarrow {\mathbb{B}}$ & The function $f$ with domain ${\mathbb{A}}$ and range ${\mathbb{B}}$\\
$\displaystyle f \circ g $ & Composition of the functions $f$ and $g$ \\
  $\displaystyle f({\bm{x}} ; {\bm{\theta}}) $ & A function of ${\bm{x}}$ parametrized by ${\bm{\theta}}$.
  (Sometimes we write $f({\bm{x}})$ and omit the argument ${\bm{\theta}}$ to lighten notation) \\
$\displaystyle \log x$ & Natural logarithm of $x$ \\
$\displaystyle \sigma(x)$ & Logistic sigmoid, $\displaystyle \frac{1} {1 + \exp(-x)}$ \\
$\displaystyle \zeta(x)$ & Softplus, $\log(1 + \exp(x))$ \\
$\displaystyle || {\bm{x}} ||_p $ & $L^p$ norm of ${\bm{x}}$ \\
$\displaystyle || {\bm{x}} || $ & $L^2$ norm of ${\bm{x}}$ \\
$\displaystyle x^+$ & Positive part of $x$, i.e., $\max(0,x)$\\
$\displaystyle \bm{1}_\mathrm{condition}$ & is 1 if the condition is true, 0 otherwise\\
\end{tabular}
\egroup
\vspace{0.25cm}

\section{Final instructions}
Do not change any aspects of the formatting parameters in the style files.
In particular, do not modify the width or length of the rectangle the text
should fit into, and do not change font sizes (except perhaps in the
\textsc{References} section; see below). Please note that pages should be
numbered.

\section{Preparing PostScript or PDF files}

Please prepare PostScript or PDF files with paper size ``US Letter'', and
not, for example, ``A4''. The -t
letter option on dvips will produce US Letter files.

Consider directly generating PDF files using \verb+pdflatex+
(especially if you are a MiKTeX user).
PDF figures must be substituted for EPS figures, however.

Otherwise, please generate your PostScript and PDF files with the following commands:
\begin{verbatim}
dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps
ps2pdf mypaper.ps mypaper.pdf
\end{verbatim}

\subsection{Margins in LaTeX}

Most of the margin problems come from figures positioned by hand using
\verb+\special+ or other commands. We suggest using the command
\verb+\includegraphics+
from the graphicx package. Always specify the figure width as a multiple of
the line width as in the example below using .eps graphics
\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.eps}
\end{verbatim}
or
\begin{verbatim}
   \usepackage[pdftex]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.pdf}
\end{verbatim}
for .pdf graphics.
See section~4.4 in the graphics bundle documentation (\url{http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps})

A number of width problems arise when LaTeX cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \verb+\-+ command.

\subsubsection*{Author Contributions}
If you'd like to, you may include  a section for author contributions as is done
in many journals. This is optional and at the discretion of the authors.

\subsubsection*{Acknowledgments}
Use unnumbered third level headings for the acknowledgments. All
acknowledgments, including those to funding agencies, go at the end of the paper.

redpajama_set_name: RedPajamaArXiv