% !TEX root = ../../main.tex
\section{Classic diffusion theory in population genetics}

The pioneers of the modern-synthesis of evolution established solid foundations
for how to understand the different forces at play that shape the structure of
evolving populations. But combining all of these forces into a single
description remained challenging. In other words, describing the outcome of an
evolutionary process that combined deterministic processes such as selection
and mutation with the stochasticity of evolution presented a challenging
mathematical puzzle. The first attempt to put everything together came from
Sewall Wright himself, but the achievement of formalizing the theory is often
associated with a remarkable Japanese population genetics by the name of Motoo
Kimura.

Kimura jump to fame when in 1968 he presented his \textit{Neutral theory of
molecular evolution}. In here Kimura argued that associating every biological
aspect to the outcome of a selection-driven process was too much of a naive
picture. We previously mentioned that Wright thought about genetic drift as an
important component of the evolutionary process, but in this 1968 paper Kimura
argued that it was \textit{the} dominant force in evolution.

But what really brings us here to study part of Kimura's scientific legacy is
his use of diffusion equations in the context of population genetics. What we
know now as Kimura's diffusion theory is nothing else than the application of
the mathematical tools developed for the study of physical diffusion of
particles to evolving populations. Diffusion is one of several transcendent
concepts in physics that demonstrate the power of universal concepts. The same
equation that describes how the concentration of a chemical species changes as
a function of space and time is used to describe how the temperature profile of
an object changes as a function of space and time as well. Furthermore, as we
will study in this section, the same mathematical language can describe
diffusion on an abstract space of allele frequency. So as Joe Blitzstein likes
to say: ``nouns change, but the verbs remain the same.''

The mathematical language to study diffusion theory, as with many other field
of science is the language of probability theory. As Jaynes' famous book title
suggested, \textit{probability theory is the logic of science}. It is through
this formal language that we can assess how likely are specific events to
happen. For example, later on we'll see that within the framework of diffusion
theory is possible to ask what is the probability of an allele being fixed in
the population given the evolutionary forces acting on it. In particular we are
interested in a type of probability object known as a Markov process that we
will now define.

\subsection{Markov processes}
Of particular interested for our endeavor are the specific mathematical objects
known as Markov processes. The Markovian property - named after Andrey Markov,
a famous Russian mathematician from the XIX century - can be stated as follows:
A Markov process is a type of stochastic process for which the transition
states only depends on the current state.

To gain intuition for what this means imagine we are measuring a variable $x$
over time such that we have a bunch of pairs of the form $(x_1, t_1), (x_2,
t_2), (x_3, t_3), \ldots (x_{n+1}, t_{n+1})$. For a Markov process, the
probability of ending at state $x_{n+1}$ at time $t_{n+1}$ given all the
previous visited states is of the form
\begin{equation}
  P(x_{n+1}, t_{n+1} \mid x_1, t_1; x_2, t_2; \ldots; x_n, t_n) =
  P(x_{n+1}, t_{n+1} \mid x_n, t_n).
  \label{eq_chapman_kolmogorov}
\end{equation}
In other words, knowing the entire history of variable $x$ as it evolves over
time does not help us predict the next time step. All we need to know is the
last position. Markov processes are usually called memoryless stochastic
process because of this property that the process doesn't remember the full
trajectory when it ``decides'' where to move in the next step. Assuming a
Markovian process is arguably one of the most widely used assumptions in all of
science; from chemical reactions, to brownian motion of a particle, to our
particular case of interest of allele frequencies changing over time.

In the classic formulation of diffusion theory we are concerned with a
particular type of Markov processes. Diffusion theory works in the limit of
large populations where we can assume that the frequency of a particular allele
rather than being a discrete entity that changes in factors of $1 / N$ where
$N$ is the number of organisms, is a continuous value in the range between 0
and 1. We also assume that we can compute changes in this frequency in
continuous time. That means that both $f$ the frequency of an allele, and $t$
the time are continuous variables. Therefore we are interested in a type of
Markov processes that go by the creative name of continuous-time
continuous-state Markov processes. We will now state a very important equation
for this type of Markovian processes known as the Chapman-Kolmogorov equation.

\subsubsection{Chapman-Kolmogorov equation} \label{sec_chapman_kolmogorov}

The Chapman-Kolmogorov equation is an important property of continuous-time
continuous-state Markov processes that we will use for many of our derivations
in the coming sections. The equation is stated as follows: For three time points
$t_1 < t_2 < t_3$ for which we measured a stochastic variable $x$ we have that
\begin{equation}
  P(x_3, t_3 \mid x_1, t_1) = \int dx_2\; P(x_3, t_3 \mid x_2, t_2)
                                          P(x_2, t_2 \mid x_1, t_1),
\end{equation}
where the integral is taken over the domain of values that the random variable
$x$ can take. In other words, to calculate the transition probability between
$x_1$ and $x_3$ with an intermediary step $x_2$, we must add (integrate for
continuous variables) all possible values that $x_2$ can take. This concept is
schematically represented in \fref{fig_chapman_kolmogorov}

\begin{figure}[h!]
	\centering \includegraphics[width=0.5\textwidth]
  {../../fig/classic_diffusion/03_02_01_chapman_kolmogorov.png}
	\caption{\textbf{Schematic of the Chapman-Kolmogorov equation}. The
  Chapman-Kolmogorov equation is a statement about the transition between two
  points $x_1$ and $x_3$, adding all possible intermediary steps $x_2$.}
  \label{fig_chapman_kolmogorov}
\end{figure}

In the coming section we will use the Chapman-Kolmogorov equation to derive the
so-called continuous master equation. This will be the foundation from which we
will get to the main results of diffusion theory. But before jumping into such
matter we need to introduce a specific subtype of Markov processes.

\subsubsection{Stationary Markov Process}\label{sec_stationary_process}

When thinking about evolution we often summon the concept of a mapping between
genotype (sequence level description) to phenotype (observable characteristic)
to fitness (reproductive success). This concept goes by several names, but in
1932 Sewall Wright attempted to formalize this idea as \textit{adaptive
landscapes}. The phenotype itself can be a function of the genotype and the
environmental state that can itself change over time either deterministically
or stochastically. Therefore the idea of a ``landscape'' is not necessarily the
best description. Some authors have discussed concepts such as \textit{fitness
seascapes} when the environment evolves over time independently of the organism
or \textit{fitness snowscapes} when there is a direct interaction between
organisms modifying the environment, and the environment modifying the
organisms.

For simplicity we usually work with fixed fitness landscapes. This
approximation is valid under certain time-scale regimes. For example, if the
environment does change over time, but it does it on a time-scale much longer
than what it takes for organisms to adapt, then we can pretend for our
calculations that the environment is fixed, simplifying the math enormously.
For this particular case where the mapping all the way from genotype to fitness
is not a function of time is described by a type of stochastic chain known as
\textbf{stationary Markov process}. The formal definition of a stationary
Markov process can be stated as: A stationary Markov process is a memoryless
process for which when computing the moments of the distribution, these are not
affected by a time shift $\tau$, i.e.
\begin{equation}
  \ee{X(t_1 + \tau) X(t_2 + \tau) \cdots X(t_n + \tau)} =
  \ee{X(t_1) X(t_2) \cdots X(t_n)}.
\end{equation}
In other words, for a stationary Markov process what matters are the absolute
time differences $|t_2 - t_1|$ rather than the absolute time points. This
implies that the probability of our random variable having a particular value
$x$ is governed by a probability distribution $P_X(x)$ with no time dependence.
Notice the difference between the statements: On the one hand if we want to
know what is the probability of the random variable transitioning from a value
$x$ to a value $x'$ all we need to know is how much time passed between both
instances of the random process. On the other hand if we are only interested in
knowing what is the probability of the random variable taking a specific value
there is no time dependence for a stationary process since this probability
will not change over time. For the evolution questions we will be addressing
here is easy to see how this implies a fixed adaptive landscape. Since the
environment is fixed the mapping between genotype all the way to fitness will
be the same regardless of the time at which the population acquires a specific
allele frequency.

Now we are ready to derive the master equation that will describe the time
evolution of the probability distribution of allele frequencies.


\documentclass[twocolumn]{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{float}
\usepackage{bm}
\usepackage{wrapfig}
\usepackage{hyperref}
\usepackage[
backend=biber,
style=nature,
sorting=none
]{biblatex}
\addbibresource{biblio.bib}

\newcommand{\EE}{\mathit{EE}}
\newcommand{\EI}{\mathit{EI}}
\newcommand{\IE}{\mathit{IE}}
\newcommand{\II}{\mathit{II}}
\newcommand{\XY}{\mathit{XY}}
\newcommand{\set}{\mathit{set}}
\newcommand{\ext}{\mathit{ext}}
\newcommand{\up}{\mathit{up}}
\newcommand{\qss}{\mathit{qss}}
\newcommand{\De}{\mathit{Det}}
\newcommand{\Tr}{\mathit{Tr}}
\newcommand{\RE}{\mathrm{Re}}

\renewcommand{\thefigure}{S\arabic{figure}}
\setcounter{figure}{0}
%\renewcommand{\thetable}{S\Roman{table}}
%\setcounter{table}{0}
\setcounter{tocdepth}{2}

\begin{document}

\title{\vspace{-2.0cm}Supplementary Material\\{\bf Orchestrated Excitatory and Inhibitory Learning Rules Lead to the Unsupervised Emergence of Up-states and Balanced Network Dynamics}}
\author{Soldado-Magraner, Laje \& Buonomano 2021}
\date{\today}

\maketitle

\tableofcontents

\section{Summary of results}

In this section we express results in terms of the ``free'' weights $W_{\EE}$ and $W_{\IE}$, switch between weights as needed, and make use of the stability conditions of the neural subsystem. How and why all this can be done is explained in detail in Section \ref{sec.Details}.

\subsection{{\em Homeostatic} learning rule}

In continuous-time dynamics, the equations for the Homeostatic learning rule are
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha_{\EE} \, E (E_{\set} - E) \\
\frac{dW_{\EI}}{dt} & = -\alpha_{\EI} \, I (E_{\set} - E) \\
\frac{dW_{\IE}}{dt} & = +\alpha_{\IE} \, E (I_{\set} - I) \\
\frac{dW_{\II}}{dt} & = -\alpha_{\II} \, I (I_{\set} - I)
\end{aligned}
\label{eq.SH_summary}
\end{equation}

\noindent The condition for the Up state to be stable under this rule is:
\begin{equation}
\begin{aligned}
(E_{\set}^2 \alpha_{\EE} & + I_{\set}^2 \alpha_{\EI}) E_{\set} g_E W_{\IE\up} \\
> & + (E_{\set}^2 \alpha_{\IE} + I_{\set}^2 \alpha_{\II}) I_{\set}(W_{\EE\up} g_E - 1) \\
& + (E_{\set}^2 \alpha_{\EE} + I_{\set}^2 \alpha_{\EI}) \Theta_I g_E
\end{aligned}
\label{eq.SH_stable_cond_v1_summary}
\end{equation}

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{SH_stability}
\includegraphics[width=\linewidth]{SH_stability_slowInh.pdf}
\caption{Regions of stability, {\em Homeostatic} learning rule. (Top) For biologically backed parameter values and learning rates of the same value ($\alpha_{\XY} = 0.02$), the stability region of the Homeostatic learning rule has little overlap with the region where the neural subsystem is stable. (Bottom) Setting $\alpha_{\EE}=\alpha_{\EI}=0.02$ and $\alpha_{\IE}=\alpha_{\II}=0.0002$ enlarges the stability region of the learning rule and makes it overlap with the stability region of the neural subsystem.
Every label is on the side where the corresponding condition holds (synaptic stable: Eq.\ \ref{eq.SH_stable_cond_v1_summary}; detcond stable: Eq.\ \ref{eq.detcond_orig}; trcond stable: Eq.\ \ref{eq.tracecond_orig}; positive $W_{\EI}$: Eq.\ \ref{eq.positive_WEI}; positive $W_{\II}$: Eq.\ \ref{eq.positive_WII}; paradoxical: Eq.\ \ref{eq.paradoxical}).}
\label{fig.SH_stability}
\end{figure}

It is hard to determine whether the stability condition Eq.\ \ref{eq.SH_stable_cond_v1_summary} is satisfied for a general set of parameter values (see numerical analysis below). However, this condition can be re-expressed in a more useful form in terms of $W_{\EE}$ and $W_{\II}$:
\begin{equation}
\begin{aligned}
(R^2 \alpha_3 + & \alpha_4)(W_{\EE\up} \, g_E  - 1)g_I \\
& < (R^2 + \alpha_2)(W_{\II\up} \, g_I + 1)g_E
\end{aligned}
\label{eq.SH_stable_cond_v2_summary}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
R & = E_{\set}/I_{\set} \\
\alpha_2 & = \alpha_{\EI}/\alpha_{\EE} \\
\alpha_3 & = \alpha_{\IE}/\alpha_{\EE} \\
\alpha_4 & = \alpha_{\II}/\alpha_{\EE}
\end{aligned}
\end{displaymath}

\noindent Note that learning rate values of the same order lead to $\alpha_{2,3,4} \sim 1$ and that biologically backed parameter values satisfy:
\begin{displaymath}
\begin{aligned}
I_{\set} & > E_{\set} \\
g_I & > g_E \\
\end{aligned}
\end{displaymath}

\noindent both likely preventing the condition to hold. On the other hand, if $\alpha_{\IE}$ and $\alpha_{\II}$ are small enough (slow dynamics of the weights onto the inhibitory neuron) the rule can be stable. See the step-by-step derivation of this stability condition in Section \ref{sec.SH_details}.

As an illustration of the results above, in Figure \ref{fig.SH_stability}(top) we plot the stability condition Eq.\ \ref{eq.SH_stable_cond_v1_summary} with parameter values as in Table \ref{tab.params} and learning rates $\alpha_{\XY} = 0.02$. It is clear that the learning rule is stable in a region with little overlap with the stability region of the neural subsystem. The stability region can be enlarged by making the dynamics of the weights onto the inhibitory neuron slower, as in Figure \ref{fig.SH_stability}(bottom) where $\alpha_{\EE}=\alpha_{\EI}=0.02$ and $\alpha_{\IE}=\alpha_{\II}=0.0002$.

See Section \ref{sec.SH_details} for a detailed analysis.

\subsection{Homeo-antiHomeo variations}

The stability condition in the previous section was obtained by assuming all learning rates are positive. Interestingly, if some of them are negative then the Up state may still be stable. A negative learning rate can be interpreted as the corresponding equation being {\em anti}-homeostatic, i.e.\ if the neural activity ($E$ or $I$) departs from its setpoint then the rule will drive it even farther away. While this kind of behavior would be usually deemed undesired, it is worth considering due to its relationship with the paradoxical regime.

In this section we consider the Homeostatic rule, Eq.\ \ref{eq.SH_equation}, and let the learning rates $\alpha_{\XY}$ be either positive or negative. The particular case where all learning rates are positive corresponds to the original Homeostatic learning rule.

Once we free the signs of the learning rates, the Up state needs two conditions to be stable:
\begin{eqnarray}
& \begin{aligned}
(R^2 \alpha_3 + & \alpha_4)(W_{\EE\up} \, g_E  - 1)g_I \\
& < (R^2 + \alpha_2)(W_{\II\up} \, g_I + 1)g_E
\end{aligned}
\label{eq.HaH_stable_cond1} \\
& \begin{aligned}
(R^2 \alpha_3 + & \alpha_4)(R^2 + \alpha_2) > 0
\end{aligned}
\label{eq.HaH_stable_cond2}
\end{eqnarray}

\noindent where
\begin{displaymath}
\begin{aligned}
R & = E_{\set}/I_{\set} \\
\alpha_2 & = \alpha_{\EI}/\alpha_{\EE} \\
\alpha_3 & = \alpha_{\IE}/\alpha_{\EE} \\
\alpha_4 & = \alpha_{\II}/\alpha_{\EE}
\end{aligned}
\end{displaymath}

\noindent Eq.\ \ref{eq.HaH_stable_cond1} is equal to the stability condition of the original Homeostatic rule (Eq.\ \ref{eq.SH_stable_cond_v2_summary}). The additional condition Eq.\ \ref{eq.HaH_stable_cond2} is very interesting in that it allows the Up state to be stable, for instance, under a full anti-Homeo learning rule where all four learning rates are negative (leading to $\alpha_{2,3,4}$ all positive).

See details in the corresponding section of the SageMath-Jupyter notebook:

\noindent {\tt upstates-Homeostatic stability.ipynb}

\subsection{{\em CrossHomeostatic} learning rule}

In continuous-time dynamics, the equations for the CrossHomeostatic learning rule are
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha_{\EE} E (I_{\set} - I) \\
\frac{dW_{\EI}}{dt} & = -\alpha_{\EI} I (I_{\set} - I) \\
\frac{dW_{\IE}}{dt} & = -\alpha_{\IE} E (E_{\set} - E) \\
\frac{dW_{\II}}{dt} & = +\alpha_{\II} I (E_{\set} - E)
\end{aligned}
\label{eq.CH_equation}
\end{equation}

\noindent and its stability condition in terms of the free weights $W_{\EE}$ and $W_{\IE}$ reads:
\begin{equation}
\begin{aligned}
(E_{\set}^2 \alpha_{\EE} & + I_{\set}^2 \alpha_{\EI}) I_{\set} g_E W_{\IE\up} \\
& >
-(E_{\set}^2 \alpha_{\IE} + I_{\set}^2 \alpha_{\II}) E_{\set} g_E W_{\EE\up} \\
& +
(E_{\set}^2 \alpha_{\IE} + I_{\set}^2 \alpha_{\II}) (\Theta_E g_E + E_{\set})
\end{aligned}
\label{eq.CH_stable_cond_v1}
\end{equation}

\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{CH_stability}
\caption{Stability of the {\em CrossHomeostatic} rule. The rule is stable for any parameter value; the Up state is thus stable where the neural subsystem is stable, i.e.\ in the upper right region between the two green lines. Every label is on the side where the corresponding condition  holds (detcond stable: Eq.\ \ref{eq.detcond_orig}; trcond stable: Eq.\ \ref{eq.tracecond_orig}; positive $W_{\EI}$: Eq.\ \ref{eq.positive_WEI}; positive $W_{\II}$: Eq.\ \ref{eq.positive_WII}; paradoxical: Eq.\ \ref{eq.paradoxical}).}
\label{fig.CH_stability}
\end{figure}

This stability condition can be put in a simpler form by switching to $W_{\EI}$ and $W_{\IE}$:
\begin{equation}
(R^2 \alpha_3 + \alpha_4) W_{\EI\up} + (R^2 + \alpha_2) W_{\IE\up} > 0
\label{eq.CH_stable_cond_v2}
\end{equation}

\noindent (where $R$ and $\alpha_{2,3,4}$ are defined as in the previous subsection). This condition is always satisfied because the weights and parameters are positive definite and thus the rule is stable for any choice of parameter values (as long as the neural subsystem is). Fig.\ \ref{fig.CH_stability} shows the stability region of the neural subsystem for the set of parameter values of Table \ref{tab.params}. Any choice of values for the weights $W_{\EE}$ and $W_{\IE}$ within the stability region of the neural subsystem will lead to a stable Up state.

See Section \ref{sec.otherRules_details} for a detailed analysis.

\subsection{{\em TwoTerm} learning rule}

\begin{figure*}[!ht]
\centering
\parbox[c]{\columnwidth}{
\includegraphics[width=\columnwidth]{TT_stability_slowHomeo.pdf}
\includegraphics[width=\columnwidth]{TT_stability_slowCross.pdf}}
\parbox[c]{\columnwidth}{
\begin{figure}[H]
\includegraphics[width=\columnwidth]{TT_stability_equal.pdf}
\caption{Regions of stability, {\em TwoTerm} rule. (Top Left) $\alpha=0.02$, $\beta=0.005$. (Top Right) $\alpha=0.02$, $\beta=0.02$. (Bottom Left) $\alpha=0.0002$, $\beta=0.02$. Every label is on the side where the  corresponding  condition  holds (synaptic stable: Eq.\ \ref{eq.TT_stable_cond_v1}; detcond stable: Eq.\ \ref{eq.detcond_orig}; trcond stable: Eq.\ \ref{eq.tracecond_orig}; positive $W_{\EI}$: Eq.\ \ref{eq.positive_WEI}; positive $W_{\II}$: Eq.\ \ref{eq.positive_WII}; paradoxical: Eq.\ \ref{eq.paradoxical}).}
\label{fig.TT_stability}
\end{figure}}
\end{figure*}

The equations for the TwoTerm learning rule in continuous-time dynamics are
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha E (I_{\set} - I) + \beta E (E_{\set} - E) \\
\frac{dW_{\EI}}{dt} & = -\alpha I (I_{\set} - I) - \beta I (E_{\set} - E) \\
\frac{dW_{\IE}}{dt} & = -\alpha E (E_{\set} - E) + \beta E (I_{\set} - I) \\
\frac{dW_{\II}}{dt} & = +\alpha I (E_{\set} - E) - \beta I (I_{\set} - I)
\end{aligned}
\label{eq.TT_summary}
\end{equation}

\noindent and its stability condition in terms of the free weights $W_{\EE}$ and $W_{\IE}$ is
\begin{equation}
\begin{aligned}
(I_{\set} \alpha & + E_{\set} \beta) g_E W_{\IE\up} \\
& > (I_{\set} \beta - E_{\set} \alpha) g_E W_{\EE\up} \\
& + (\Theta_E g_E + E_{\set}) \alpha + (\Theta_I g_E - I_{\set}) \beta
\end{aligned}
\label{eq.TT_stable_cond_v1}
\end{equation}

In Figure \ref{fig.TT_stability} we plot the stability condition of this rule, Eq.\ \ref{eq.TT_stable_cond_v1}, for three different parameter values: $\alpha \gg \beta$ (the ``CrossHomeostatic'' terms dominate over the ``Homeostatic'' terms, and the rule is stable with the largest stability region); $\alpha = \beta$ (the two terms are of comparable size); and $\alpha \ll \beta$ (the ``Homeostatic'' terms dominate instead, and the stability region of the rule is as small as that of the Homeostatic learning rule).

In order to determine the validity of the stability condition, Eq.\ \ref{eq.TT_stable_cond_v1}, in a more general situation, we rewrite it in a more useful form:
\begin{equation}
(a - b)\beta < (a^\prime + b^\prime + c) \alpha
\label{eq.TT_stable_cond_v2}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
a & = (W_{\EE\up}g_E - 1) E_{\set} I_{\set} g_I \\
a^\prime & = (W_{\EE\up}g_E - 1)E_{\set}^2 g_I \\
b & = (W_{\II\up}g_I + 1) E_{\set} I_{\set} g_E \\
b^\prime & = (W_{\II\up}g_I + 1)I_{\set}^2 g_E \\
c & = (I_{\set}\Theta_I - E_{\set}\Theta_E)g_E g_I
\end{aligned}
\end{displaymath}

\noindent Note that the following is satisfied for a biologically backed set of parameter values:
\begin{displaymath}
\begin{aligned}
I_{\set} & > E_{\set} \\
\Theta_I & > \Theta_E
\end{aligned}
\end{displaymath}

\noindent and thus it is likely that $c>0$. In addition, $b$ and $b^\prime$ are positive definite, and $a,a^\prime>0$ in the paradoxical regime ($W_{\EE}g_E - 1 > 0$). All this makes the stability condition likely satisfied, and thus the learning rule stable. Finally, a small enough $\beta$ would make the condition more likely to hold.

\begin{figure*}[!ht]
\centering
\parbox[c]{\columnwidth}{
\includegraphics[width=\columnwidth]{SS_stability.pdf}
\includegraphics[width=\columnwidth]{SS_stability_slowExc.pdf}}
\parbox[c]{\columnwidth}{
\begin{figure}[H]
\includegraphics[width=\columnwidth]{SS_stability_slowInh.pdf}
\caption{Regions of stability, {\em SynapticScaling} rule. (Top Left) Equal rates ($\alpha_{\XY}=0.02$). (Top Right) Slow inhibitory ($\alpha_{\EE,\EI}=0.02$, $\alpha_{\IE,\II}=0.002$). (Bottom Left) Slow excitatory ($\alpha_{\EE,\EI}=0.002$, $\alpha_{\IE,\II}=0.02$). Every label is on the side where the  corresponding  condition  holds (synaptic stable: Eq.\ \ref{eq.SS_stable_cond_v1} after switching to $W_{\EE}$ and $W_{\IE}$; detcond stable: Eq.\ \ref{eq.detcond_orig}; trcond stable: Eq.\ \ref{eq.tracecond_orig}; positive $W_{\EI}$: Eq.\ \ref{eq.positive_WEI}; positive $W_{\II}$: Eq.\ \ref{eq.positive_WII}; paradoxical: Eq.\ \ref{eq.paradoxical}).}
\label{fig.SS_stability}
\end{figure}}
\end{figure*}

See Section \ref{sec.otherRules_details} for a detailed analysis.

\subsection{{\em SynapticScaling} learning rule}

The equations for the SynapticScaling learning rule in continuous-time dynamics are
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha_{\EE} (E_{\set} - E) W_{\EE} \\
\frac{dW_{\EI}}{dt} & = -\alpha_{\EI} (E_{\set} - E) W_{\EI} \\
\frac{dW_{\IE}}{dt} & = +\alpha_{\IE} (I_{\set} - I) W_{\IE} \\
\frac{dW_{\II}}{dt} & = -\alpha_{\II} (I_{\set} - I) W_{\II}
\end{aligned}
\label{eq.SS_summary}
\end{equation}

\noindent and the condition for the Up state to be stable under this rule is
\begin{equation}
(W_{\EE\up}g_E - 1)a < (W_{\II\up}g_I + 1)b
\label{eq.SS_stable_cond_v1}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
a = & \, (I_{\set} W_{\II} \alpha_4 + \Theta_I \alpha_3)g_I \\
b = & \, E_{\set\up} W_{\EE\up}g_E \\
& + ((W_{\EE\up} g_E - 1) E_{\set} - \Theta_E g_E)\alpha_2 \\
& - (W_{\EE\up} g_E - 1) I_{\set} \alpha_3
\end{aligned}
\end{displaymath}

\noindent (where $\alpha_{2,3,4}$ are defined as in previous subsections). This stability condition does not hold for biologically backed parameter values unless the dynamics of the weights onto the inhibitory neuron are slow enough (and in a few fine-tuned cases). To show this, we express the stability condition in terms of the free weights $W_{\EE}$ and $W_{\IE}$ and plot it with parameter values as in Table \ref{tab.params} and equal rates ($\alpha_{\XY}=0.02$; Figure \ref{fig.SS_stability} top left). The stability condition is a homographic function (i.e.\ a hyperbola) with stability regions in its upper-left and lower-right quadrants---entirely outside the stability region of the neural subsystem. If the dynamics of the weights onto the excitatory neuron are made slower, the homographic function is even steeper (bottom left); if the weights onto the inhibitory neuron are made slower instead, the stability regions switch and overlap with the stability region of the neural subsystem, making the Up state stable (top left).

It is illustrative to consider the particular case where all learning rates are equal. In this case the stability condition, Eq.\ \ref{eq.SS_stable_cond_v1}, doesn't depend on the learning rates and takes the simpler form:
\begin{equation}
\begin{aligned}
(W_{\II\up} g_I + & 1)a > (W_{\EE\up} g_E - 1)a^\prime \\
& + (W_{\EE\up} g_E - 1)(W_{\II\up} g_I + 1)b
\end{aligned}
\label{eq.SS_stable_cond_v2}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
a & = (E_{\set} W_{\EE\up} - \Theta_E)g_E \\
a^\prime & = (I_{\set} W_{\II\up} + \Theta_I)g_I \\
b & = I_{\set} - E_{\set}
\end{aligned}
\end{displaymath}

\noindent Note that in a biologically backed set of parameter values the following is true:
\begin{displaymath}
\begin{aligned}
I_{\set} & > E_{\set} \\
g_I & > g_E \\
\Theta_I & > \Theta_E
\end{aligned}
\end{displaymath}

\noindent This makes $b>0$ and likely $a^\prime>a$ (in addition, $a^\prime$ is a sum of positive terms while $a$ is a difference). Then in the paradoxical regime ($W_{\EE}g_E - 1 > 0$) it seems likely that the stability condition is not satisfied, because the right-hand side is a sum of positive terms and one of them is likely greater than the left-hand side. The SynapticScaling rule is then likely unstable when the learning rates are equal.

A more general case with different learning rates can be analyzed by grouping terms in the following way:
\begin{displaymath}
\begin{aligned}
(I_{\set} W_{\II\up} \alpha_4 & + \Theta_I\alpha_3) g_I(W_{\EE\up} g_E - 1) \\
< & (((W_{\EE\up} g_E - 1) E_{\set} - \Theta_E g_E)\alpha_2 \\
& -(W_{\EE\up} g_E - 1) I_{\set}\alpha_3 \\
& + E_{\set} W_{\EE\up} g_E) (W_{\II\up} g_I + 1)
\end{aligned}
\label{eq.SS_stable_cond_v3}
\end{displaymath}

\noindent If $(W_{EE}g_E - 1) > 0$ (paradoxical regime), then decreasing $\alpha_3$ and/or $\alpha_4$ (slow dynamics of the weights onto the inhibitory neuron) helps satisfying the condition and thus making the rule stable.

See Section \ref{sec.otherRules_details} for a detailed analysis.

\subsection{{\em ForcedBalance} learning rule}

The equations for the ForcedBalance learning rule are
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha_1 g_E E(E_{\set} - E) \\
\frac{dW_{\EI}}{dt} & = \frac{1}{\tau_0} (W_{\EI\up} - W_{\EI}) \\
\frac{dW_{\IE}}{dt} & = +\alpha_3 g_I E(I_{\set} - I) \\
\frac{dW_{\II}}{dt} & = \frac{1}{\tau_0} (W_{\II\up} - W_{\II})
\end{aligned}
\label{eq.FB_summary}
\end{equation}

\noindent and the conditions for the Up state to be stable under this rule are
\begin{equation}
\begin{aligned}
a_1 + b_1(W_{\II\up} \, g_I + 1) & < b_1^\prime(W_{\EE\up} \, g_E - 1) \\
a_2 + b_2(W_{\II\up} \, g_I + 1) & < b_2^\prime(W_{\EE\up} \, g_E - 1)
\end{aligned}
\label{eq.FB_stable_cond}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
a_1 & = (I_{\set} \Theta_E \Theta_I \, \alpha_1 \, g_E g_I + E_{\set}^3 \alpha_3) \, g_E g_I \\
b_1 & = I_{\set}^2 \Theta_E \, \alpha_1 g_E^2 g_I - E_{\set}^2 I_{\set} \, \alpha_1 \, g_E^2 \\
b_1^\prime & = E_{\set} I_{\set} \Theta_I \, \alpha_1 \, g_E g_I^2 + E_{\set}^2 I_{\set} \, \alpha_3 \, g_I^2 \\
a_2 & = 2 \Theta_E \Theta_I \, \alpha_1 \, g_E^2 g_I^2 \\
b_2 & = 2I_{\set} \Theta_E \, \alpha_1 \, g_E^2 g_I - E_{\set}^2 \, \alpha_1 \, g_E^2 \\
b_2^\prime & = 2E_{\set} \Theta_I \, \alpha_1 \, g_E g_I^2 + E_{\set}^2 \, \alpha_3 \, g_I^2
\end{aligned}
\end{displaymath}

In Figure \ref{fig.FB_stability} we plot the stability condition of this rule, Eq.\ \ref{eq.FB_stable_cond}, for three different parameter values: $\alpha_1 = \alpha_3$, $\alpha_1 \gg \alpha_3$ (inhibitory plasticity slower); and $\alpha_1 \ll \alpha_3$ (excitatory plasticity slower).

\begin{figure*}[!ht]
\centering
\parbox[c]{\columnwidth}{
\includegraphics[width=\columnwidth]{FB_stability_equal.pdf}
\includegraphics[width=\columnwidth]{FB_stability_sloweralpha3.pdf}}
\parbox[c]{\columnwidth}{
\begin{figure}[H]
\includegraphics[width=\columnwidth]{FB_stability_fasteralpha3.pdf}
\caption{Regions of stability, {\em ForcedBalance} rule. (Top Left) Equal rates ($\alpha_{1,3}=0.02$). (Top Right) Slow inhibitory ($\alpha_1=0.02$, $\alpha_3=0.0002$). (Bottom Left) Slow excitatory ($\alpha_1=0.0002$, $\alpha_3=0.02$). Every label is on the side where the  corresponding  condition  holds (synaptic stable 1 and 2: Eq.\ \ref{eq.FB_stable_cond} after switching to $W_{\EE}$ and $W_{\IE}$; detcond stable: Eq.\ \ref{eq.detcond_orig}; trcond stable: Eq.\ \ref{eq.tracecond_orig}; positive $W_{\EI}$: Eq.\ \ref{eq.positive_WEI}; positive $W_{\II}$: Eq.\ \ref{eq.positive_WII}; paradoxical: Eq.\ \ref{eq.paradoxical}).}
\label{fig.FB_stability}
\end{figure}}
\end{figure*}

In order to decide whether conditions Eq.\ \ref{eq.FB_stable_cond} are satisfied in a more general case, note that $b_1$ and $b_2$ on the left-hand side are subtractions whereas $b_1^\prime$ and $b_2^\prime$ on the right-hand side are sums of positive definite terms, which helps satisfying the condition. On the other hand, one of the stability conditions of the neural subsystem might counter the effect: $(W_{\II\up} \, g_I + 1)\tau_E > (W_{\EE\up} \, g_E - 1)\tau_I$ (see Section \ref{sec.neural} below) but for biologically backed parameter values it is $\tau_E > \tau_I$ thus leaving room for the condition to hold. See Section \ref{sec.otherRules_details} for a detailed analysis.

\section{Detailed calculations}
\label{sec.Details}

\subsection{Overview}

We analyze the whole neural+synaptic system for every synaptic learning rule considered in this work, and study their stability. In every case, the general prescription is:
\begin{enumerate}
\item Take the combined neural+synaptic system and nondimensionalize all variables \cite[see Sections 1.2 and 1.4 of Ref.\ ][]{Keener1998}\cite[see Section 3.5 of Ref.\ ][]{Strogatz2018}, so that the two different time scales are evident (fast neural, slow synaptic).
\item Make a quasi-steady state (QSS) approximation of the neural subsystem \cite{Keener1998,Strogatz2018}. This means we will consider the neural subsystem is fast enough so that it converges ``instantaneously'' (when compared to the synaptic subsystem) to its corresponding fixed point.
For this we will require that the stability conditions of the neural subsystem are satisfied (see below).
\item Find the steady-state solution of the synaptic subsystem, i.e.\ the Up state fixed point; compute the Jacobian of the synaptic subsystem at the Up state; compute the eigenvalues of the Jacobian \cite{Strogatz2018,Wiggins1996}. Two out of the four eigenvalues are expected to be zero because the Up state is not an isolated fixed point of the system but a continuous 2D plane in 4D weight space.
\item Address (linear) stability. If both nonzero eigenvalues have negative real part, then the Up state is stable under the learning rule; if at least one of the nonzero eigenvalues has positive real part, then the Up state is unstable \cite{Strogatz2018,Wiggins1996}. (A note on abuse of notation: we might say indistinctly ``the Up state is stable/unstable'' and ``the learning rule is stable/unstable''.)
\end{enumerate}

Eigenvalues and stability in the presence of continuous, i.e. non-isolated, attractors have been discussed in the context of neural networks for eye position control \cite{Seung1996,Seung1998} (keep in mind that the eigenvalues' critical value in these references is 1 instead of zero because they consider eigenvalues of the connectivity matrix alone, whereas we consider eigenvalues of the whole linear part). As the Up state is a collection of non-isolated fixed points that form a 2D plane, there is no dynamics along the plane, and the linear stability analysis is enough to fully address stability---we do have two zero eigenvalues, but there is no need to compute the center manifold \cite{Wiggins1996} because the other two eigenvalues represent the whole dynamics around the fixed point and have nonzero real part.

In order to apply the tools from Dynamical Systems' theory for flows in a unified way for both the neural and synaptic subsystems, we will switch from a discrete-time description of synaptic weight dynamics (where the change in weight $W$ is represented by $\Delta W$ applied every certain time interval) to a continuous-time description (where the weights are continuously evolving albeit with a long time scale $\tau_0$):
\begin{displaymath}
\Delta W \rightarrow \tau_0 \frac{dW}{dt}
\end{displaymath}

In the following we first define the neural subsystem and compute its stability conditions (next subsection). Then we consider every learning rule in detail (following subsections).

\subparagraph{Paradoxical regime.} In this text we show detailed calculations of the stability conditions for the Homeostatic learning rule in the paradoxical regime only; see Section \ref{sec.nonparadoxical} for the non-paradoxical case.

\subsection{Neural dynamics}
\label{sec.neural}

For the neural+synaptic system in the QSS approximation to be stable under a specific synaptic learning rule, it is necessary that the neural subsystem is stable so it remains in its QSS solution as the weights evolve. In this section we define the neural subsystem and compute its stability conditions.

(SageMath code in the Supplementary Material: {\tt upstates-Neural subsystem stability.ipynb})

\subsubsection{System's equations and fixed points}

We consider a two-subpopulation model with firing-rate units $E$ and $I$ with ReLU activation functions (gain $g_X$, threshold $\Theta_X$, with $X=E,I$). The dynamics for synaptic currents above threshold is given by:
\begin{equation}
\begin{aligned}
\frac{dE}{dt} & = \frac{1}{\tau_E}(-E + g_E(W_{\EE}E - W_{\EI}I - \Theta_E)) \\
\frac{dI}{dt} & = \frac{1}{\tau_I}(-I + g_I(W_{\IE}E - W_{\II}I - \Theta_I))
\end{aligned}
\label{eq.neural}
\end{equation}

\noindent All variables and parameters are definite positive. In this subsection the synaptic weights $W_{\mathit{XY}}$ are fixed.

\subparagraph{Up state.} The Up state is the non-trivial fixed point of the system (i.e.\ a steady-state solution where $dE/dt=dI/dt=0$):
\begin{equation}
\begin{aligned}
E_{\up} & = (W_{\EI} \, g_I \, \Theta_I - (W_{\II} \, g_I + 1) \, \Theta_E) \, g_E / C \\
I_{\up} & = ((W_{\EE} \, g_E - 1) \, \Theta_I - W_{\IE} \, g_E \, \Theta_E) \, g_I / C
\end{aligned}
\label{eq.upstate_orig}
\end{equation}

\noindent where
\begin{equation}
C = W_{\EI} W_{\IE} \, g_E\, g_I - (W_{\II} \, g_I + 1)(W_{\EE} \, g_E - 1)
\label{eq.den}
\end{equation}

The activity of the excitatory and inhibitory subpopulations at the Up state, $E_{\up}$ and $I_{\up}$, depend on all weight values. Only some of the combinations, however, lead to a stable steady state. We compute the stability conditions in the following subsection.

\subsubsection{Stability of neural fixed point (Up state)}

The Jacobian matrix, that is the matrix of first derivatives, gives information regarding the stability of fixed points: if the real parts of its eigenvalues are all negative, then the fixed point is stable.

The Jacobian of the neural system (Eq.\ \ref{eq.neural}) is
\begin{equation}
J = \left(\begin{array}{rr}
(W_{\EE} g_E - 1)/\tau_E & -W_{\EI} g_E/\tau_E \\
W_{\IE} g_I/\tau_I & -(W_{\II} g_I + 1)/\tau_I
\end{array}\right)
\end{equation}

\noindent Its eigenvalues can be expressed as:
\begin{equation}
\lambda_{1,2} = \frac{1}{2}\left(\Tr \pm \sqrt{\Tr^2 - 4\De} \right)
\label{eq.neural.eigvals}
\end{equation}

\noindent where $\Tr$ and $\De$ are the trace and determinant of the matrix, respectively. For eigenvalues either complex or purely real, their real parts are negative (and thus the Up state is stable) when $\De>0$ and $\Tr<0$, that is:
\begin{eqnarray}
& W_{\EI} W_{\IE} g_E g_I > (W_{\EE} g_E - 1)(W_{\II} g_I + 1) \label{eq.detcond_orig}\\
& (W_{\II} g_I + 1)\tau_E > (W_{\EE} g_E - 1)\tau_I
\label{eq.tracecond_orig}
\end{eqnarray}

\noindent Note that the positive determinant condition, Eq.\ \ref{eq.detcond_orig}, is equivalent to $C>0$ (Eq.\ \ref{eq.den}).

In the following, we will require that the stability conditions of the neural subsystem, Eqs.\ \ref{eq.detcond_orig} and \ref{eq.tracecond_orig}, are satisfied.

\subsubsection{Weight values consistent with an Up state}

The Up state relationships, Eq.\ \ref{eq.upstate_orig}, are expressed as the $E$ and $I$ values resulting from a given set of weight values. If we set instead $E$ and $I$ to their target values $E_{\set}$ and $I_{\set}$ and solve for the weights, we get the weight values that are consistent with a given Up state activity:
\begin{equation}
\begin{aligned}
W_{\EI\up} & = \frac{(E_{\set} W_{\EE\up} - \Theta_E) \, g_E - E_{\set}}{I_{\set} \, g_E} \\
W_{\II\up} & = \frac{(E_{\set} W_{\IE\up} - \Theta_I) \, g_I - I_{\set}}{I_{\set} \, g_I}
\end{aligned}
\label{eq.upstate_W}
\end{equation}

\noindent Note first that any stable learning rule for the evolution of the weights for the neural subsystem (Eq.\ \ref{eq.neural}) must converge to weight values in accordance with these relationships (either in the form Eq.\ \ref{eq.upstate_W} or Eq.\ \ref{eq.upstate_orig}).

Second, note that the system is underdetermined and that is why two of the weights are free (chosen to be $W_{\EE}$ and $W_{\EI}$). Note also that all weight values must be positive; specifically, requiring $W_{\EI\up}>0$ and $W_{\II\up}>0$ leads to
\begin{eqnarray}
W_{\EE\up} & > & \frac{\Theta_E \, g_E + E_{\set}}{E_{\set} \, g_E}
\label{eq.positive_WEI} \\
W_{\IE\up} & > & \frac{\Theta_I \, g_I + I_{\set}}{E_{\set} \, g_I}
\label{eq.positive_WII}
\end{eqnarray}

\noindent We refer to these expressions as the ``positive $W_{\EI}$'' and the ``positive $W_{\II}$'' conditions, respectively.

\subsubsection{Paradoxical effect}

The paradoxical effect arises when an external depolarization of the inhibitory subpopulation (increase of $I$) produces an actual {\em decrease} of $I$. In this model, an external depolarization of $I$ can be mimicked by a decrease of its threshold $\Theta_I$, thus there is a paradoxical effect whenever the coefficient of $\Theta_I$ in the numerator of $I_{up}$ is positive. The coefficient is $g_I \, (W_{\EE} \, g_E - 1)/C$ and thus there is paradoxical effect if
\begin{equation}
W_{\EE} \, g_E - 1 >0
\label{eq.paradoxical}
\end{equation}

The paradoxical effect can also be seen in a plot of the Up-state values $E_{\up}$ and $I_{\up}$ (Eq.\ \ref{eq.upstate_orig}) as a function of each individual weight. Specifically, from a naive point of view $I_{\up}$ should increase when $W_{\IE}$ is increased, and decrease when $W_{\II}$ is increased; however, it does the opposite in either case (see Figure \ref{fig.paradoxical}).

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{paradoxical_E}
\includegraphics[width=\linewidth]{paradoxical_I}
\caption{Paradoxical effect in the neural subsystem ($W_{\EE}=5$; parameter values as in Table \ref{tab.params}). $E_{\up}$ behaves as expected when each weight is varied. $I_{\up}$, however, shows paradoxical behavior when either $W_{\IE}$ or $W_{\II}$ are varied. Dashed lines are the vertical asymptote of every case.}
\label{fig.paradoxical}
\end{figure}

\begin{table}[ht]
\centering
\begin{tabular}{lcrclcr}
$I_{\set}$ & = & 14 & \hspace{1cm} & $E_{\set}$ & = & 5 \\
$g_I$ & = & 4 & & $g_E$ & = & 1 \\
$\Theta_I$ & = & 25 & & $\Theta_E$ & = & 4.8 \\
$\tau_I$ & = & 2 & & $\tau_E$ & = & 10
\end{tabular}
\caption{Parameter values throughout the Supplementary Material. This set of parameter values makes the neural subsystem to be in the paradoxical regime (i.e.\ the Up state is an inhibition-stabilized fixed point \cite{Sadeh2021}). For non-paradoxical conditions, see Section \ref{sec.nonparadoxical}.}
\label{tab.params}
\end{table}

\subsection{{\em Homeostatic} learning rule: Detailed calculation}
\label{sec.SH_details}

In this section we show in detail the calculation of the stability condition for the Homeostatic learning rule.

(SageMath code in the Supplementary Material: {\tt upstates-Homeostatic stability.ipynb})

\subsubsection{Definition of the learning rule}

In continuous-time dynamics, the Homeostatic learning rule reads:
\begin{equation}
\begin{aligned}
\frac{dW_{\EE}}{dt} & = +\alpha_{\EE} \, E (E_{\set} - E) \\
\frac{dW_{\EI}}{dt} & = -\alpha_{\EI} \, I (E_{\set} - E) \\
\frac{dW_{\IE}}{dt} & = +\alpha_{\IE} \, E (I_{\set} - I) \\
\frac{dW_{\II}}{dt} & = -\alpha_{\II} \, I (I_{\set} - I)
\end{aligned}
\label{eq.SH_equation}
\end{equation}

\noindent where $\alpha_{\XY}$ $(X,Y=E,I)$ are the learning rates (with appropriate units) setting the time scales of the weight dynamics, and $E_{\set}$ and $I_{\set}$ are the set points of the excitatory and inhibitory subpopulations, respectively.

The fixed points of the system (i.e.\ steady states) are determined by setting all derivatives to zero. There is a non-trivial fixed point compatible with the neural subsystem being above threshold: the Up state, that is the set of weight values such that:
\begin{equation}
\begin{aligned}
E_{\up} & = E_{\set} \\
I_{\up} & = I_{\set}
\end{aligned}
\label{eq.SH_upstate_set}
\end{equation}

\noindent The values of the weights corresponding to the Up state are given by the (underdetermined) system defined by equating Eqs.\ \ref{eq.SH_upstate_set} and \ref{eq.upstate_orig}. Since it is a two-equation system for a set of four unknown weights, there are two free weights that we choose to be $W_{\EE\up}$ and $W_{\IE\up}$. The values of the other two are given by Eq.\ \ref{eq.upstate_W}. This means that the Up-state fixed point is actually a continuous set of non-isolated fixed points forming a 2D plane in 4D weight space. In other words, there is an infinite number of weight values compatible with the Up state (possibly not all stable, though).

\subsubsection{Nondimensionalization}

Next we nondimensionalize all variables in order to have a simpler system and make the QSS approximation in a safe way. We define new (nondimensional) variables $e$, $i$, $\tau$, $w_{\EE}$, $w_{\EI}$, $w_{\IE}$, and $w_{\II}$, and their corresponding scaling parameters. We substitute the new variables into the full system (neural+synaptic, Eqs.\ \ref{eq.neural} and \ref{eq.SH_equation}) and choose the values of the scaling parameters such that all nondimensional variables are of order 1 (see attached SageMath code). With this, the full system reads:
\begin{equation}
\begin{aligned}
\epsilon_E \frac{de}{d\tau} & = -e + R e w_{\EE} - \frac{i w_{\EI}}{R} - \theta_E \\
\epsilon_I \frac{di}{d\tau} & = -i + \frac{R e w_{\IE}}{g} - \frac{i w_{\II}}{Rg} - \theta_I \\
\frac{dw_{\EE}}{d\tau} & = -e(e-1) \\
\frac{dw_{\EI}}{d\tau} & = +\alpha_2 i(e-1) \\
\frac{dw_{\IE}}{d\tau} & = -\alpha_3 e(i-1) \\
\frac{dw_{\II}}{d\tau} & = +\alpha_4 i(i-1)
\end{aligned}
\label{eq.SH_full_norm}
\end{equation}

\noindent where we defined the new parameters
\begin{displaymath}
\begin{aligned}
\epsilon_E & = \tau_E/\tau_0 \\
\epsilon_I & = \tau_I/\tau_0 \\
\tau_0 & = 1/(\alpha g_E E_{\set} I_{\set}) \\
R & = E_{\set}/I_{\set} \\
g & = g_E/g_I \\
\alpha_2 & = \alpha_{\EI}/\alpha_{\EE} \\
\alpha_3 & = \alpha_{\IE}/\alpha_{\EE} \\
\alpha_4 & = \alpha_{\II}/\alpha_{\EE} \\
\theta_E & = (g_E/E_{\set}) \Theta_E \\
\theta_I & = (g_I/I_{\set}) \Theta_I
\end{aligned}
\end{displaymath}

\subsubsection{Quasi-steady state approximation}

Neural dynamics evolves in a much shorter time scale ($\tau_E$ and $\tau_I$) than synaptic dynamics ($\tau_0$):
\begin{displaymath}
\begin{aligned}
\tau_E & \ll \tau_0 \implies \epsilon_E \ll 1 \\
\tau_I & \ll \tau_0 \implies \epsilon_E \ll 1
\end{aligned}
\end{displaymath}

\noindent which implies
\begin{equation}
\begin{aligned}
\epsilon_E\frac{de}{d\tau} & \sim 0 \\
\epsilon_I\frac{di}{d\tau} & \sim 0
\end{aligned}
\label{eq.qss_approx}
\end{equation}

\noindent thus we can safely assume $e$ and $i$ very quickly reach quasi-equilibrium values, i.e.\ practically instantaneous convergence to quasi-steady state (QSS) values as if the weights were fixed, while the synaptic weights evolve according to their slow dynamics. This allows us to reduce the system's dimensionality from six to four.

In the QSS approximation, the values of the nondimensionalized excitatory and inhibitory activities instantaneously track the slow dynamics of the learning rule. They are determined by applying Eq.\ \ref{eq.qss_approx} to the first two rows of Eq.\ \ref{eq.SH_full_norm}; solving for $e$ and $i$ leads to
\begin{equation}
\begin{aligned}
e_{\qss} & = (g \theta_I w_{\EI} - (w_{\II} + Rg) \theta_E)/c \\
i_{\qss} & = (R g \theta_I (R w_{\EE} - 1) - R^2 \theta_E w_{\IE})/c
\end{aligned}
\label{eq.SH_upstate_qss}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
c = R w_{\EI} w_{\IE} - (w_{\II} + Rg)(R w_{\EE} - 1)
\end{aligned}
\end{displaymath}

The full system in the QSS approximation reads
\begin{equation}
\begin{aligned}
\frac{dw_{\EE}}{d\tau} & = -e_{\qss}(e_{\qss}-1) \\
\frac{dw_{\EI}}{d\tau} & = +\alpha_2 i_{\qss}(e_{\qss}-1) \\
\frac{dw_{\IE}}{d\tau} & = -\alpha_3 e_{\qss}(i_{\qss}-1) \\
\frac{dw_{\II}}{d\tau} & = +\alpha_4 i_{\qss}(i_{\qss}-1)
\end{aligned}
\label{eq.SH_full_norm_qss}
\end{equation}

\noindent where $e_{\qss}$ and $i_{\qss}$ are nonlinear functions of the weights as defined by Eq.\ \ref{eq.SH_upstate_qss}.

Note that the Up state fixed point, defined by making all derivatives equal to zero, can be expressed as
\begin{equation}
\begin{aligned}
e_{\qss} & = 1 \\
i_{\qss} & = 1
\end{aligned}
\label{eq.SH_upstate_ei}
\end{equation}

\noindent which is the nondimensionalized version of Eq.\ \ref{eq.SH_upstate_set}. The weight values compatible with this condition are defined by equating Eqs.\ \ref{eq.SH_upstate_qss} and \ref{eq.SH_upstate_ei}:
\begin{equation}
\begin{aligned}
w_{\EI\up} & = R (Rw_{\EE\up} - 1) - R \theta_E \\
w_{\II\up} & = R (Rw_{\IE\up} - g) - Rg \theta_I
\end{aligned}
\label{eq.SH_upstate_w}
\end{equation}

\noindent ($w_{\EE\up}$ and $w_{\IE\up}$ are free). This is the nondimensionalized version of Eq.\ \ref{eq.upstate_W}.

\subsubsection{Stability condition}

The program for assessing linear stability of the Up state is as follows: a) compute the Jacobian (the matrix of first derivatives) of Eq.\ \ref{eq.SH_full_norm_qss} and evaluate it at the Up state; b) compute the eigenvalues of the Jacobian (two of them will be zero because the fixed points form a continuous 2D plane in phase space); c) If the real part of the two nonzero eigenvalues is negative then the Up state is stable; if at least one of the nonzero eigenvalue has positive real part then the Up state is unstable.

\paragraph{Jacobian matrix.}

Let the full system in the QSS approximation (Eq.\ \ref{eq.SH_full_norm_qss}) be written as
\begin{displaymath}
\begin{aligned}
\frac{dw_{\EE}}{d\tau} & = f_{\EE}(e_{\qss},i_{\qss}) \\
\frac{dw_{\EI}}{d\tau} & = f_{\EI}(e_{\qss},i_{\qss}) \\
& \mbox{etc}\ldots
\end{aligned}
\end{displaymath}

\noindent where $e_{\qss}$ and $i_{\qss}$ are functions of the weights as defined by Eq.\ \ref{eq.SH_upstate_qss}. By applying the chain rule the elements $J_{ij}$ ($i,j=1\ldots4$) of the Jacobian matrix can be expressed as
\begin{displaymath}
\begin{aligned}
J_{11} & = \frac{df_{\EE}}{dw_{\EE}} = \frac{df_{\EE}}{de_{\qss}} \frac{de_{\qss}}{dw_{\EE}} + \frac{df_{\EE}}{di_{\qss}} \frac{di_{\qss}}{dw_{\EE}} \\
J_{12} & = \frac{df_{\EE}}{dw_{\EI}} = \frac{df_{\EE}}{de_{\qss}} \frac{de_{\qss}}{dw_{\EI}} + \frac{df_{\EE}}{di_{\qss}} \frac{di_{\qss}}{dw_{\EI}} \\
J_{13} & = \ldots \\
J_{21} & = \frac{df_{\EI}}{dw_{\EE}} = \frac{df_{\EI}}{de_{\qss}} \frac{de_{\qss}}{dw_{\EE}} + \frac{df_{\EI}}{di_{\qss}} \frac{di_{\qss}}{dw_{\EE}} \\
J_{22} & = \ldots \\
& \mbox{etc}\ldots
\end{aligned}
\end{displaymath}

In order to have the Jacobian specialized in the Up state, these expressions are to be substituted by Eqs.\ \ref{eq.SH_upstate_qss}-\ref{eq.SH_upstate_w}.

\paragraph{Eigenvalues of the Jacobian matrix.}

The Jacobian matrix has two zero eigenvalues and two nonzero eigenvalues. The nonzero eigenvalues have the form:
\begin{equation}
\lambda_{\pm} = \frac{A \pm \sqrt{A^2 - DC}}{C}
\label{eq.SH_eigvals}
\end{equation}

\noindent where
\begin{equation}
\begin{aligned}
A & = R^2 g \theta_I + (R^2 \alpha_3 + \alpha_4) R w_{\EE\up} \\
& - (R^2 + \alpha_2)R w_{\IE\up} + \alpha_2 g \theta_I - R^2 \alpha_3 - \alpha_4 \\
C & = 2R(R g \theta_I w_{\EE\up} - R \theta_E w_{\IE\up} - g \theta_I) \\
D & = 2(R^2\alpha_3 + \alpha_4)(R^2 + \alpha_2)/R
\end{aligned}
\end{equation}

\paragraph{Sign of the eigenvalues.}

To determine the sign of the real part of Eq.\ \ref{eq.SH_eigvals}, first note that the factor $D$ is positive definite. Second, $C$ must be positive because it is related to one of the stability conditions of the neural subsystem (Eq.\ \ref{eq.detcond_orig}, after substituting back to dimensionalized quantities). Note next that $A^2 - DC$ is less than $A^2$ (since $C$ and $D$ are positive), and thus the square root is either real and less than $|A|$ or imaginary, both cases leading to $\RE(A \pm \sqrt{A^2-DC}) < 0$ if $A<0$. The learning rule is then stable (both eigenvalues have negative real part) if $A<0$, which in terms of the original parameters and free weights $W_{\EE}$ and $W_{\IE}$ reads:
\begin{equation}
\begin{aligned}
(E_{\set}^2 \alpha_{\EE} & + I_{\set}^2 \alpha_{\EI}) E_{\set} g_E W_{\IE\up} \\
> & + (E_{\set}^2 \alpha_{\IE} + I_{\set}^2 \alpha_{\II}) I_{\set}(W_{\EE\up} g_E - 1) \\
& + (E_{\set}^2 \alpha_{\EE} + I_{\set}^2 \alpha_{\EI}) \Theta_I g_E
\end{aligned}
\label{eq.SH_stable_cond_v1}
\end{equation}

\subsubsection{Analysis of the stability condition}

It is hard to determine whether the stability condition Eq.\ \ref{eq.SH_stable_cond_v1} is satisfied for a general set of parameter values (see numerical analysis below). However, by using the Up state relationship Eq.\ \ref{eq.upstate_W}, this condition can be re-expressed in a more useful form in terms of $W_{\EE}$ and $W_{\II}$:
\begin{equation}
\begin{aligned}
(R^2 \alpha_3 + & \alpha_4)(W_{\EE\up} \, g_E  - 1)g_I \\
& < (R^2 + \alpha_2)(W_{\II\up} \, g_I + 1)g_E
\end{aligned}
\label{eq.SH_stable_cond_v2}
\end{equation}

Note that learning rates values of the same order lead to $\alpha_{2,3,4} \sim 1$ and that biologically backed parameter values satisfy:
\begin{displaymath}
\begin{aligned}
I_{\set} & > E_{\set} \\
g_I & > g_E \\
\end{aligned}
\end{displaymath}

\noindent both likely preventing the condition to hold.

On the other hand, small enough values of $\alpha_3$ and $\alpha_4$ (by making the dynamics of the weights onto the inhibitory neuron $W_{\IE}$ and $W_{\II}$ slower) would help satisfy the condition thus making the system stable.

\subsubsection{Relationship between the stability condition and the paradoxical condition}

The boundary of the stability condition for this learning rule, Eq.\ \ref{eq.SH_stable_cond_v1}, is a linear function in the $(W_{\EE},W_{\IE})$ space with a slope that tends to infinity as the excitatory learning rates ($\alpha_{\EE,\EI}$) tend to zero:
\begin{displaymath}
\mbox{slope} = \frac{(E_{\set}^2 \alpha_{\IE} + I_{\set}^2\alpha_{\II})I_{\set}}{(E_{\set}^2\alpha_{\EE} + I_{\set}^2\alpha_{\EI})E_{\set}}
\end{displaymath}

\noindent while its root is a complicated expression (see SageMath notebook) that tends to $W_{\EE} = 1/g_E$. The region of stability is to the left of the line. Thus, the boundary of stability in this limit coincides exactly with the boundary of the paradoxical condition ($W_{\EE}>1/g_E$). This can be construed as an inconsistency/contradiction between the stability of the rule and the existence of the paradoxical effect.

\subsection{Detailed calculations for the other rules}
\label{sec.otherRules_details}

The stability calculations for the rest of the rules follow very similar paths. They can be found in the corresponding SageMath-Jupyter notebooks:
\begin{itemize}
\item[] {\tt upstates-CrossHomeostatic stability.ipynb}
\item[] {\tt upstates-TwoTerm stability.ipynb}
\item[] {\tt upstates-SynapticScaling stability.ipynb}
\item[] {\tt upstates-ForcedBalance stability.ipynb}
\end{itemize}

\subsection{Stability of the rules in a non-paradoxical regime}
\label{sec.nonparadoxical}

All results above were developed with the neural subsystem set in the paradoxical regime---that is, the region in ($W_{\EE},W_{\IE}$) leading to a stable Up state was completely within the paradoxical region ($W_{\EE}g_E > 1$). In order to show the importance of the paradoxical behavior for the stability of the learning rules, we also computed the stability conditions of every learning rule in a more general setting where the excitatory subpopulation in the neural subsystem has an external, constant, excitatory input current $I_{\ext}$. This allows the neural subsystem to display both paradoxical and non-paradoxical stable behavior (in the second case, at the expense of the Up state not being an inhibition-stabilized fixed point). The results can be found in the following SageMath-Jupyter notebooks:
\begin{itemize}
\item[] {\tt upstates-Neural subsystem stability-with Iext.ipynb}
\item[] {\tt upstates-Homeostatic stability-with Iext.ipynb}
\item[] {\tt upstates-TwoTerm stability-with Iext.ipynb}
\item[] {\tt upstates-SynapticScaling stability-with Iext.ipynb}
\item[] {\tt upstates-ForcedBalance stability-with Iext.ipynb}
\end{itemize}

\noindent (The stability condition of the CrossHomeostatic learning rule doesn't depend on $I_{\ext}$.)

\section{Learning rule from loss function}

(SageMath code in the Supplementary Material: {\tt upstates-Loss function.ipynb})

Here we show how to compute the learning rule starting from a loss function. Then we make an approximation by considering the weight values are small, and take that as input to interpret several learning rules that don't come from a loss function.

\subsection{General prescription}

We consider the full neural+synaptic system in the QSS approximation (see e.g.\ Section \ref{sec.SH_details}). In this approximation the neural subsystem is represented by the quasi-steady-state values
\begin{equation}
\begin{aligned}
E & = E_{\up}(W_{\EE},W_{\EI},W_{\IE},W_{\II}) \\
I & = I_{\up}(W_{\EE},W_{\EI},W_{\IE},W_{\II})
\label{eq.LF_EIup}
\end{aligned}
\end{equation}

\noindent where the functions $E_{\up}$ and $I_{\up}$ are defined by Eq.\ \ref{eq.upstate_orig} (see \cite{Mackwood2020} for a related discussion on quasi-steady state, synaptic plasticity, and gradient descent).

The synaptic subsystem, that is the learning rule, will be obtained as a result of considering a specific loss function, and the general prescription to compute the learning rule from a loss function $L$ is the following:
\begin{enumerate}
\item Consider a loss function depending on $E$ and $I$ (which in turn depend on all weights):
\begin{displaymath}
L = L(E,I)
\end{displaymath}

\noindent Conditions to be satisfied by the loss function are, for instance, to be smooth enough (i.e.\ continuous and differentiable) and to have a minimum when the activities $E$ and $I$ are at the set points $E_{\set}$ and $I_{\set}$ (i.e.\ homeostatic plasticity).

\item The dynamics of the weights is such that it follows a gradient descent on the loss function towards its minimum. In vector notation:
\begin{equation}
\Delta \mathbf{W} = -\alpha \nabla L
\label{eq.LF_gradient}
\end{equation}

\noindent with a single learning rate $\alpha$ for simplicity. The unfolded learning rules, that is the equations that govern the weights' dynamics, are then
\begin{equation}
\begin{aligned}
\Delta W_{\EE} & = - \alpha \frac{\partial L}{\partial W_{\EE}} \\
\Delta W_{\EI} & = - \alpha \frac{\partial L}{\partial W_{\EI}} \\
\Delta W_{\IE} & = - \alpha \frac{\partial L}{\partial W_{\IE}} \\
\Delta W_{\II} & = - \alpha \frac{\partial L}{\partial W_{\II}} \end{aligned}
\label{eq.LF_generic_rule}
\end{equation}

\item The partial derivatives of the loss function in Eq.\ \ref{eq.LF_generic_rule} are:
\begin{equation}
\begin{aligned}
\frac{\partial L}{\partial W_{\EE}} & = \frac{\partial L}{\partial E} \frac{\partial E}{\partial W_{\EE}} + \frac{\partial L}{\partial I} \frac{\partial I}{\partial W_{\EE}} \\
\frac{\partial L}{\partial W_{\EI}} & = \frac{\partial L}{\partial E} \frac{\partial E}{\partial W_{\EI}} + \frac{\partial L}{\partial I} \frac{\partial I}{\partial W_{\EI}} \\
\frac{\partial L}{\partial W_{\IE}} & = \frac{\partial L}{\partial E} \frac{\partial E}{\partial W_{\IE}} + \frac{\partial L}{\partial I} \frac{\partial I}{\partial W_{\IE}} \\
\frac{\partial L}{\partial W_{\II}} & = \frac{\partial L}{\partial E} \frac{\partial E}{\partial W_{\II}} + \frac{\partial L}{\partial I} \frac{\partial I}{\partial W_{\II}} \end{aligned}
\label{eq.LF_chain_rule}
\end{equation}

\noindent or, in vector notation:
\begin{equation}
\nabla L = \frac{\partial L}{\partial E} \nabla E + \frac{\partial L}{\partial I} \nabla I
\label{eq.LF_partial}
\end{equation}

\noindent Here we use the chain rule for the derivatives because it gives us much more compact expressions at the end.

\item The partial derivatives in the gradients $\displaystyle \nabla E = \left(\frac{\partial E}{\partial W_{EE}}, \ldots \right)$ and $\displaystyle \nabla I = \left( \frac{\partial I}{\partial W_{EE}}, \ldots \right)$ etc. are to be taken from the quasi-steady-state values of $E$ and $I$, Eq.\ \ref{eq.LF_EIup}. We will, however, compute the partial derivatives from the implicit expressions given by setting $dE/dt=dI/dt=0$ in Eq.\ \ref{eq.neural} without solving for $E$ and $I$.
\end{enumerate}

\subsection{Detailed calculation}

\subsubsection{Exact learning rules}

\paragraph{Loss function.}

We choose a very general loss function that depends homeostatically on both $E$ and $I$ activities:
\begin{equation}
L(E,I) = \frac{1}{2}(E_{set} - E)^2 + \frac{1}{2}(I_{set} - I)^2
\label{eq.LF_energy}
\end{equation}

\noindent This loss function is an elliptic paraboloid in $(E,I)$ space with a global minimum at $(E_{\set},I_{\set})$ so a gradient descend learning rule as above should converge to that minimum (see Liapunov function and gradient systems: \cite[Section 1.1B]{Wiggins1996}\cite[Sections 9.3 and 9.4]{Hirsch1974}\cite[Section 7.2]{Strogatz2018}). Keep in mind, however, that $L$ has a different shape when expressed as a function of the weights, and that $E$ and $I$ are not necessarily monotonic functions of the weights (particularly for a paradoxical system), so the conditions for the set point of $L$ to be stable or a global minimum or even unique are not necessarily satisfied.

\paragraph{Partial derivatives of $L$.}

The partial derivatives of $L$ with respect to $E$ and $I$ are simply
\begin{equation}
\begin{aligned}
\frac{\partial L}{\partial E} & = - (E_{\set} - E) \\
\frac{\partial L}{\partial I} & = - (I_{\set} - I)
\end{aligned}
\label{eq.LF_dLdEI}
\end{equation}

\paragraph{Partial derivatives of $E$ and $I$.}

We compute the partial derivatives $\partial X/\partial W_{\XY}$ ($X,Y = E,I$) by first equating the neural subsystem (Eq.\ \ref{eq.neural}) to zero:
\begin{equation}
\begin{aligned}
E & = g_E(W_{\EE}E - W_{\EI}I - \Theta_E)) \\
I & = g_I(W_{\IE}E - W_{\II}I - \Theta_I))
\end{aligned}
\label{eq.LF_implicit_functions}
\end{equation}

\noindent then differentiating the implicit functions:
\begin{equation}
\begin{aligned}
\frac{\partial E}{\partial W_{\EE}} & = g_E(E + W_{\EE} \frac{\partial E}{\partial W_{\EE}}) - g_E W_{\EI} \frac{\partial I}{\partial W_{\EE}} \\
\frac{\partial E}{\partial W_{\EI}} & = g_E W_{\EE} \frac{\partial E}{\partial W_{\EI}} - g_E (I + W_{\EI} \frac{\partial I}{\partial W_{\EI}}) \\
\frac{\partial E}{\partial W_{\IE}} & = g_E W_{\EE} \frac{\partial E}{\partial W_{\IE}} - g_E W_{\EI}\frac{\partial I}{\partial W_{\IE}} \\
\frac{\partial E}{\partial W_{\II}} & = g_E W_{\EE} \frac{\partial E}{\partial W_{\II}} - g_E W_{\EI} \frac{\partial I}{\partial W_{\II}} \\
\frac{\partial I}{\partial W_{\EE}} & = g_I W_{\IE} \frac{\partial E}{\partial W_{\EE}} - g_I W_{\II} \frac{\partial I}{\partial W_{\EE}} \\
\frac{\partial I}{\partial W_{\EI}} & = g_I W_{\IE} \frac{\partial E}{\partial W_{\EI}} - g_I W_{\II} \frac{\partial I}{\partial W_{\EI}} \\
\frac{\partial I}{\partial W_{\IE}} & = g_I (E + W_{\IE} \frac{\partial E}{\partial W_{\IE}}) - g_I W_{\II} \frac{\partial I}{\partial W_{\IE}} \\
\frac{\partial I}{\partial W_{\II}} & = g_I W_{\IE} \frac{\partial E}{\partial W_{\II}} - g_I (I + W_{\II} \frac{\partial I}{\partial W_{\II}})
\end{aligned}
\label{eq.LF_implicit_derivs}
\end{equation}

\noindent and then solving for the derivatives:
\begin{equation}
\begin{aligned}
\frac{\partial E}{\partial W_{\EE}} & = -(E W_{\II} \, g_E \, g_I + E g_E)/C \\
\frac{\partial E}{\partial W_{\EI}} & = (I W_{\II} \, g_E \,g_I + I g_E)/C \\
\frac{\partial E}{\partial W_{\IE}} & = E W_{\EI} \, g_E \, g_I/C \\
\frac{\partial E}{\partial W_{\II}} & = -I W_{\EI} \, g_E \, g_I/C \\
\frac{\partial I}{\partial W_{\EE}} & = -E W_{\IE} \, g_E \, g_I/C \\
\frac{\partial I}{\partial W_{\EI}} & = I W_{\IE} \, g_E \, g_I/C \\
\frac{\partial I}{\partial W_{\IE}} & = (E W_{\EE} \, g_E - E) g_I/C \\
\frac{\partial I}{\partial W_{\II}} & = -(I W_{\EE} \, g_E - I)g_I/C
\end{aligned}
\label{eq.LF_explicit_derivs}
\end{equation}

\noindent where
\begin{displaymath}
C = W_{\EI} W_{\IE} \, g_E\, g_I - (W_{\II} \, g_I + 1)(W_{\EE} \, g_E - 1)
\end{displaymath}

\paragraph{Exact learning rules.}

Putting everything together, the learning rules Eq.\ \ref{eq.LF_generic_rule} are:
\begin{equation}
\begin{aligned}
\Delta W_{\EE} = \, -\frac{\alpha}{C}
( & (I_{\set} - I) E W_{\IE} \, g_e \, g_I \\
& + (E_{\set} - E) E (W_{\II} \, g_I + 1)g_E) \\
\Delta W_{\EI} = \, +\frac{\alpha}{C}
( & (I_{\set} - I) I W_{\IE}\, g_e \, g_I \\
& + (E_{\set} - E) I (W_{\II} \, g_I + 1)g_E) \\
\Delta W_{\IE} = \, +\frac{\alpha}{C}
( & (E_{\set} - E) E W_{\EI}\, g_e \, g_I \\
& + (I_{\set} - I) E (W_{\EE} \, g_E - 1)g_I) \\
\Delta W_{\II} = \, -\frac{\alpha}{C}
( & (E_{\set} - E) I W_{\EI}\, g_e \, g_I \\
& + (I_{\set} - I) I (W_{\EE} \, g_E - 1)g_I)
\end{aligned}
\end{equation}

\noindent Note that these are very complicated, nonlinear expressions because both $E$ and $I$ depend on all weights via Eq.\ \ref{eq.LF_implicit_functions}. Also the denominator $C$ depends on all weights (see previous paragraph).

\subsubsection{Approximation}

We want simpler expressions for the learning rules. Note that the exact expressions above all have a homeostatic factor (either $E - E_{\set}$ or $I - I_{\set}$) and a presynaptic factor (either $E$ or $I$), while the rest are complicated expressions coming from the derivatives $\partial E/\partial W_{\XY}$ and  $\partial I/\partial W_{\XY}$. We want to keep the homeostatic and presynaptic factors as they are while simplifying the rest of the expressions (explicit dependence on the weights including $C$) by performing a lowest-order Taylor series expansion of the explicit dependence of Eqs.\ \ref{eq.LF_explicit_derivs} on the weights. Although this is not a textbook Taylor expansion of the full expressions, it is very informative because the results can be much more easily interpreted (for a similar approach see \cite{Mackwood2020}).

We perform a zeroth-order approximation of the derivatives $\partial E/\partial W_{\XY}$ and  $\partial I/\partial W_{\XY}$ as functions of the weights (i.e.\ while holding the presynaptic factors $E$ and $I$ constant) around the Up state. In this approximation the weights are not small but close to their target values, represented by the relationships Eq.\ \ref{eq.upstate_W}. By substituting the result in Eq.\ \ref{eq.LF_chain_rule}, we get the following approximated learning rules:
\begin{equation}
\begin{aligned}
\Delta W_{\EE} & = + \alpha_E E(I_{\set} - I) + \beta_E E(E_{\set} - E) \\
\Delta W_{\EI} & = - \alpha_E I(I_{\set} - I) - \beta_E I(E_{\set} - E) \\
\Delta W_{\IE} & = - \alpha_I E(E_{\set} - E) + \beta_I E(I_{\set} - I) \\
\Delta W_{\II} & = + \alpha_I I(E_{\set} - E) - \beta_I I(I_{\set} - I)
\end{aligned}
\label{eq.LF_approx}
\end{equation}

\noindent where
\begin{displaymath}
\begin{aligned}
\alpha_E & = \alpha g_E E_{\set} W_{\IE\up}/D \\
\alpha_I & = \alpha A/D \\
\beta_E & = \alpha g_E B/D \\
\beta_I & = \alpha I_{\set} (1 - W_{\EE\up} g_E)/D
\end{aligned}
\end{displaymath}

\noindent and
\begin{displaymath}
\begin{aligned}
A & = E_{\set} W_{\EE\up} g_E - \Theta_E g_E - E_{\set} \\
B & = E_{\set} W_{\IE\up} - \Theta_I \\
D & = \Theta_I W_{\EE\up} g_E - \Theta_E W_{\IE\up} g_E - \Theta_I
\end{aligned}
\end{displaymath}

\paragraph{Analysis.} Note that $\alpha_E$, $\alpha_I$, $\beta_E$, and $\beta_I$ are all constant. Furthermore, note that
\begin{itemize}
\item $A>0$ as it is equal to the ``positive $W_{\EI}$'' condition, Eq.\ \ref{eq.positive_WEI};
\item $B>0$ as it is part of the ``positive $W_{\II}$'' condition, Eq.\ \ref{eq.positive_WII};
\item $D>0$ as it is equal to the numerator of $I_{\up}$, Eq.\ \ref{eq.upstate_orig} (up to a positive factor), which must be positive because the denominator is.
\end{itemize}

\noindent Interestingly, note that the learning rate $\beta_I$ can be either negative or positive depending on whether the Up state where the dynamics is converging to is paradoxical ($W_{\EE\up}g_E - 1 > 0$) or not ($W_{\EE\up}g_E - 1 < 0$).

\noindent Note that the terms with $\alpha_{E,I}$ in the approximated learning rules, Eq.\ \ref{eq.LF_approx}, are exactly equal to the CrossHomeostatic rules, Eq.\ \ref{eq.CH_equation}. Additionaly, the terms with $\beta_{E,I}$ are exactly equal to the Homeostatic rules, Eq.\ \ref{eq.SH_equation}, unless $\beta_I<0$ which would make the learning rule a Cross-Homeo-antiHomeo hybrid.

\printbibliography

\end{document}


\section{Numerical Sequences and Series}

\subsection*{Definition 3.5}
Since $\{p_n\} \to p \implies \forall \epsilon, \: \exists N | n \geq N \implies \abs{p_n - p} < \epsilon$,
we can choose $k | n_k \geq N \implies \{p_{n_k}\} \to p$.
The reverse direction can be shown via contradiction of $\{p_n\} \to p$.

\subsection*{Examples 3.18}
(a) Density of rationals in reals.

(b) $\abs{s_n} < 1$, take $n$ odd to get -1 and even to get 1.

(c) Every subsequential limit has to converge to $s$.

\subsection*{Theorem 3.19}
For all $\{n_k\}$, we have $\exists K | k \geq K \implies  n_k \geq N \implies \lim_{k \to \infty} t_{n_k} - s_{n_k} \geq 0$.

\subsection*{Theorem 3.26}
$s_n = 1 + x + ... + x^n \implies x s_n = x + x^2 + ... + x^{n+1} \implies (1 - x) s_n = 1 - x^{n+1}$.

\subsection*{Examples 3.40}
(a) Root test: $n \to \infty$.

(b) Ratio test: $\frac{1}{n+1} \to 0$.

(c) $1 \to 1$.

(d) Ratio test: $\frac{n}{n+1} \to 1$. $z = 1$ leads to harmonic series.

(e) Ratio test: $\frac{n^2}{(n+1)^2} \to 1$.

\subsection*{Example 3.53}
$\sum_{k=1}^{\infty} \frac{1}{4k-3} + \frac{1}{4k-1} - \frac{1}{2k} < \frac{5}{6} + \sum_{k=2}^{\infty} \frac{1}{4k-4} + \frac{1}{4k-4} - \frac{1}{2k}$. The RHS converges since $\frac{1}{4k-4} + \frac{1}{4k-4} - \frac{1}{2k} = \frac{1}{2k^2 - 2k}$.

\newpage

\subsection{Exercise 1}
All we need is the inequality $\abs{s_n - s} \geq \abs{\abs{s_n} - \abs{s}}$.
The converse is not true, since we can take $s_n = (-1)^n$.

\subsection{Exercise 2}
My original idea: $\sqrt{(n + x)^2} - n = x$. Setting $(n + x)^2 \geq n^2 + n$ gives
$x^2 \geq (1 - 2x) n$. The last inequality is only true for all $n$ when $x \geq \frac{1}{2}$.
This implies that $\frac{1}{2}$ is the supremum of $\sqrt{n^2 + n} - n$. Since
$\sqrt{n^2 + n} - n$ is increasing, it converges to $\frac{1}{2}$.

Better: $(\sqrt{n^2 + n} - n) (\sqrt{n^2 + n} + n) = n \implies \sqrt{n^2 + n} - n = \frac{1}{\sqrt{1 + \frac{1}{n}} + 1}$.

\subsection{Exercise 3}
Clearly $s_{n + 1} > s_n$. We can see that $s_n < 2$ by induction, since $s_1 < 2$ and
$2 + \sqrt{s_n} < 4$. This gives that $s_n$ is monotone and bounded, implying it converges.

\subsection{Exercise 4}
\begin{align*}
&s_{2m + 1} = \sum_{i = 1}^{m} \frac{1}{2^i}, \: s_{2m} = \sum_{i = 2}^{m} \frac{1}{2^i} \\
&\implies \limsup_{n \to \infty} s_n = 1, \: \liminf_{n \to \infty} s_n = \frac{1}{2}
\end{align*}

\subsection{Exercise 5}
\begin{align*}
        \limsup_{n \to \infty} (a_n + b_n) &= \sup_{\{k\}}{\{\lim_{k \to \infty} (a_{n_k} + b_{n_k})\}} \\
                                           &= \sup_{\{k\}}{\{\lim_{k \to \infty} a_{n_k} + \lim_{k \to \infty} b_{n_k}\}}
\end{align*}

\subsection{Exercise 6}
(a) $\sqrt{n + 1} - \sqrt{n} = \frac{1}{\sqrt{n + 1} + \sqrt{n}}$ diverges from comparison
to harmonic series (same technique as Exercise 2).

(b) Converges, by comparison to $\sum_{n = 1}^{\infty} \frac{1}{n^p}$ for $p = \frac{3}{2}$.

(c) Converges by root test, since $\lim_{n \to \infty} n^{\frac{1}{n}} = 1$.

(d) Converges when $\abs{z} > 1$ and diverges otherwise. To see this, put $z = \abs{z}e^{i\theta}$
to get $\lim_{n \to \infty} \frac{1}{1 + \abs{z}^n e^{ni\theta}}$.

\subsection{Exercise 7}
We proceed via the ratio test.
\begin{align*}
        \limsup_{n \to \infty} \frac{n}{n+1} * \frac{\sqrt{a_{n + 1}}}{\sqrt{a_n}} &= \limsup_{n \to \infty} \frac{n}{n+1} \limsup_{n \to  \infty} \frac{\sqrt{a_{n+1}}}{\sqrt{a_n}} \\
                                                                                &= \sqrt{\limsup_{n \to \infty} \frac{a_{n+1}}{a_n}} \\
                                                                                &< 1
\end{align*}
Since $\sum a_n$ converges.

\subsection{Exercise 8}
Since $b_n$ is monotonic and bounded, $\abs{b_n} \leq B$ for all $n$. Then we have that $\sum a_n b_n$
converges by the comparison test, since $\abs{a_n b_n} \leq B \abs{a_n}$ and $B \sum a_n$ converges.

\subsection{Exercise 9}
(a) Applying the ratio test, we see that $\abs{z} \limsup_{n \to  \infty} \abs{\frac{(n + 1)^3}{n^3}} < 1$
when $\abs{z} < 1$. Thus $\sum n^3 z^n$ has radius of convergence 1.

(b) Again, applying the ratio test, we see that $2 \abs{z} \limsup_{n \to \infty} \abs{\frac{1}{n + 1}} = 0$,
implying $R = +\infty$.

(c) The ratio test is the only hammer we need: $2 \abs{z} \limsup_{n \to \infty} \abs{\frac{n^2}{(n+1)^2}} < 1$
gives $R = \frac{1}{2}$.

(d) What are the other tests again? $\frac{\abs{z}}{3} \limsup_{n \to \infty} \abs{\frac{(n + 1)^3}{n^3}} < 1$
gives $R = 3$.

\subsection{Exercise 10}
The infinitely many non-zero $a_n$ must satisfy $\abs{a_n} \geq 1$. The radius of convergence of $\sum a_n z^n$
will be maximized when $\abs{a_n}$ is minimized, so we can just consider the case where there are infinitely many
$\abs{a_n} = 1$. In this case, we can choose a subsequence $a_{n_k}$ consisting only of 1. Applying the ratio
test using this subsequence gives  $\abs{z} < 1$.

\subsection{Exercise 15}
Theorems 3.22, 3.23, and 3.25(a) require no changes in their proofs, since the Cauchy criterion
is applicable for $\mathbb{R}^k$. Theorem 3.33(a, b) also require no
changes once we have the comparison test for $\mathbb{R}^k$. For Theorem 3.33(c), we can take
$a \in \mathbb{R}^k$ such that all of its components are $\frac{1}{n}$ or $\frac{1}{n^2}$.

Theorem 3.34(a, b) just need to be modified to use $\frac{\abs{a_{n+1}}}{\abs{a_n}}$.
Theorem 3.42 needs to be modified to use the dot product, but then it follows from applying
the $\mathbb{R}$ version of 3.42 to the components of the dot product sum.
Theorems 3.45, 3.47, and 3.55 require no changes to their proofs.


\subsection{Additive Expression}

\begin{grammar}
\nonterminal{additive-expression}
    \produces
	\nonterminal{multiplicative-expression} \\
    \produces
	\nonterminal{additive-expression}
	\lexkeyword{+}
	\nonterminal{multiplicative-expression} \\
    \produces
	\nonterminal{additive-expression}
	\lexkeyword{-}
	\nonterminal{multiplicative-expression} \\
\end{grammar}

\subsubsection{Eliminating left recursion}

Eliminating left recursion leads to
\begin{grammar}
\underbrace{\nonterminal{additive-expression}}_{=: A}
    \produces
	\underbrace{\nonterminal{multiplicative-expression}}_{=: M}
	\underbrace{\nonterminal{additive-expression'}}_{=:A'}\\
\nonterminal{additive-expression'}
    \produces
	\lexkeyword{+}
	\nonterminal{multiplicative-expression}
	\nonterminal{additive-expression'}
	\\
    \produces
	\lexkeyword{-}
	\nonterminal{multiplicative-expression}
	\nonterminal{additive-expression'}
	\\
    \produces
\end{grammar}\\[-0.5cm]
\noindent
The left associativity of the operators \lexkeyword{+} and \lexkeyword{-} needs
to be handled manually by the parser.

\paragraph{Item automata for A}

\[
\begin{tikzpicture}[
    every text node part/.style={align=center},
    initial text =
]
    \node[state,initial]
	(S)[rounded rectangle, draw]
	{{$[A\to .MA']$}};
    \node[state]
	(M)[rounded rectangle, draw,right=of S]
	{{$[A\to M.A']$}};
    \node[state,accepting]
	(A_)[rounded rectangle, draw,right=of M]
	{{$[A\to MA'.]$}};
    \path[->] (S) edge  node [above] {M} (M);
    \path[->] (M) edge  node [above] {A'} (A_);
\end{tikzpicture}
\]

\paragraph{Item automata for A'}
\[
\begin{tikzpicture}[
    every text node part/.style={align=center},
    initial text =
]
    \node[state,initial,accepting]
	(S)[rounded rectangle, draw]
	{
	    {$[A'\to .+MA']$}\\
	    {$[A'\to .-MA']$}\\
	    {$[A'\to .]$}
	};
    \node[state]
	(Add)[rounded rectangle, draw,right=of S]
	{{$[A'\to +.MA']$}};
    \node[state]
	(Sub)[rounded rectangle, draw,below=of Add]
	{{$[A'\to -.MA']$}};
    \node[state]
	(M)[rounded rectangle, draw,right=of Add]
	{
	    {$[A'\to +M.A']$} \\
	    {$[A'\to -M.A']$}
	};
    \node[state,accepting]
	(E)[rounded rectangle, draw,right=of M]
	{
	    {$[A'\to +MA'.]$} \\
	    {$[A'\to -MA'.]$}
	};
    \path[->] (S) edge  node [above] {\lexkeyword{+}} (Add);
    \path[->] (S) edge  node [above] {\lexkeyword{-}} (Sub);
    \path[->] (Add) edge  node [above] {M} (M);
    \path[->] (Sub) edge  node [above] {M} (M);
    \path[->] (M) edge  node [above] {A'} (E);
\end{tikzpicture}
\]

\subsubsection{Eliminating left recursion (in EBNF Form)}

Expressing the BNF notation in EBNF reveals how the item automata for $A$ and
$A'$ can be combined in an implementation using a while loop
checking if the
current token is in $\mathrm{First}(A') = \{ \lexkeyword{+}, \lexkeyword{-}\}$:
\begin{grammar}
\underbrace{\nonterminal{additive-expression}}_{= A}
    \produces
	\underbrace{\nonterminal{multiplicative-expression}}_{= M}
	\underbrace{
	\{\;
	    \nonterminal{additive-op}
	    \nonterminal{multiplicative-expression}
	\}
	}_{=A'}
	\\
\nonterminal{additive-op}
    \produces
    \lexkeyword{+} \\
    \produces
    \lexkeyword{-} \\
\end{grammar}\\[-0.5cm]


%\begin{document}
\section{Generalizing the Leapfrog Integrator}%
\label{sec:generalizing_lf}
As in the HMC algorithm, we start by augmenting the current state $x \in
\mathbb{R}^n$ with a continuous momentum variable $v \in \mathbb{R}^{n}$ drawn
from a standard normal distribution.
%
Additionally, we introduce a binary direction variable $d \in \{ -1, 1\}$,
drawn from a uniform distribution.
%
The complete augmented state is then denoted by $\xi \equiv (x, v, d)$, with
probability density $p(\xi) = p(x) p(v) p(d)$.
%
To improve the overall performance of our model, for each step $t$ of the
leapfrog operator $\mathbf{L}_{\theta}$, we assign a fixed random binary mask
$m^{t} \in{\{0, 1\}}^n$ that will determine which variables are affected by
each sub-update.
%
The mask $m^t$ is drawn uniformly from the set of binary vectors satisfying
$\sum_{i=1}^{n} m_{i}^{t} = \lfloor \frac{n}{2}\rfloor$, i.e.\ half the entries
of $m^t$ are $0$ and half are $1$.
%
Additionally, we write $\bar m^{t} = \mathbbm{1} - m^{t}$ and $x_{m^t} = x
\odot m^{t}$, where $\odot$ denotes element-wise multiplication, and
$\mathbbm{1}$ the vector of $1$'s in each entry.
%

We begin with a subset of the augmented space, $\zeta_1 \equiv (x, \partial_{x}
U(x), t)$, independent of the momentum $v$.
%
We introduce three new functions of $\zeta_1$: $T_v$, $Q_v$, and $S_v$.
%
We can then perform a single time-step of our modified leapfrog integrator
$\mathbf{L}_{\theta}$.
%

First, we update the momentum $v$, which depends only on the subset $\zeta_1$.
%
This update is written
%
\begin{equation}
\label{eq:update_momentum_forward1}
  \vp = v\, \odot
      \hspace{-1mm}
      \underbrace{\exp\left(\frac{\varepsilon}{2}S_{v}(\zeta_1)\right)}_{\text{%
      \footnotesize{\textbf{Momentum scaling}\normalsize}
       }}%
      \hspace{-1mm}
       - \frac{\varepsilon}{2}\bigg[\partial_{x}U(x)\odot
      \hspace{-1mm}
      \underbrace{\exp(\varepsilon Q_{v}(\zeta_1))}_{\text{%
       \footnotesize{\textbf{Gradient scaling}\normalsize}
        }}
       + \underbrace{T_v(\zeta_1)}_{\text{%
       \footnotesize{\textbf{Translation}\normalsize}
       }}\bigg]
\end{equation}
and the corresponding Jacobian is given by:
$\exp{\left(\frac{\eps}{2}\mathbbm{1} \cdot S_{v}(\zeta_1)\right)}$.
%
Next, we update $x$ by first updating a subset of the coordinates of $x$
(determined according to the mask $m^t$), followed by the complementary subset
(determined from $\bar m^{t}$).
%
The first update affects only $x_{m^{t}}$ and produces $x^{\prime}$.
%
This update depends only on the subset $\zeta_2 \equiv (x_{\bar m^t}, v, t)$.
%
Following this, we perform the second update which only affects
$x_{\bar{m}^t}^{\prime}$ and depends only on $\zeta_3 \equiv (x_{m^t}^{\prime},
v, t)$, to produce $x^{\prime\prime}$:
%
\begin{align}
  x^{\prime} &= x_{\bar{m}^t} + m^{t}\odot\left[x \odot \exp{(\eps
      S_x(\zeta_2))} + \eps\left(v^{\prime}\odot%
    \exp{(\eps Q_x(\zeta_2))} + T_x(\zeta_2)\right)\right]\\
  x^{\prime\prime} &= x^{\prime}_{m^t} + \bar m^t \odot \left[x^{\prime} \odot
    \exp{(\eps S_x(\zeta_3))} +%
    \eps\left(v^{\prime} \odot \exp{(\eps Q_x(\zeta_3))} +
  T_x(\zeta_3)\right)\right].
\end{align}
%
with Jacobians: $\exp{(\eps m^{t} \cdot S_x(\zeta_2))}$, and
$\exp{(\eps\bar{m^t}\cdot S_x(\zeta_3))}$, respectively.
%
Finally, we proceed to update $v$ again, using the subset $\zeta_4 \equiv
(x^{\prime\prime}, \partial_{x} U^{\prime\prime}, t)$:
%
\begin{equation}
  \vpp = \vp \odot \exp\left(\frac{\varepsilon}{2} S_v(\zeta_4)\right) -
    \frac{\varepsilon}{2}\left[\partial_{x} U
  \odot \exp(\varepsilon Q_v(\zeta_4)) + T_v(\zeta_4)\right].
    \label{eq:update_momentum_forward2}
\end{equation}
%
In order to build some intuition about each of these terms, we discuss below
some of the subtleties contained in this approach and how they are (carefully)
dealt with.

The first thing to notice about these equations is that if $S_{i} = Q_{i} =
T_{i} = 0$ ($i = x, v$), we recover the previous equations for the generic
leapfrog integrator (as we would expect since we are attempting to
\emph{generalize} HMC).
%
We can also see a similarity between the equations for updating $v$ and those
for updating $x$: each update is generalized by \emph{scaling} the previous
value ($v$ or $x$), and \emph{scaling and translating} the updating value
(either $\partial_{x}\,U(x)$ or $x$).
%
It can be shown~\cite{2017arXiv171109268L}, that the scaling applied to the
momentum in Eq~\ref{eq:update_momentum_forward1} can enable, among other
things, acceleration in low-density zones to facilitate mixing between modes,
and that the scaling term applied to the gradient may allow better conditioning
of the energy landscape (e.g., by learning a diagonal inertia tensor), or
partial ignoring of the energy gradient for rapidly oscillating energies.

Second, note that because the determinant of the Jacobian appears in the
Metropolis-Hastings (MH) acceptance probability, we require the Jacobian of
each update to be efficiently computable (i.e.\ independent of the variable
actually being updated).
%
For each of the momentum updates, the input is a subset $\zeta = (x,
\partial_{x}\,U(x), t)$ of the augmented space and the associated Jacobian is
$\exp{\left(\frac{\eps}{2}\mathbbm{1}\cdot S_{v}(\zeta)\right)}$ which is
independent of $v$ as desired.
%
For the position updates however, things are complicated by the fact that the
input $\zeta$ is $x$-dependent.
%
In order to ensure that the Jacobian of the $x$ update is efficiently
computable, it is necessary to break the update into two parts following the
approach outlined in \emph{Real-valued Non-Volume Preserving transformations
(RealNVP)}~\cite{dinhRealNVP}.
%
\subsection{Metropolis-Hastings Accept/Reject}
Written in terms of these transformations, the augmented leapfrog operator
$\mathbf{L}_{\theta}$ consists of $M$ sequential applications of the
single-step leapfrog operator $\mathbf{L}_{\theta} \xi = \mathbf{L}_{\theta}(x,
v, d) = (x^{\prime\prime\times M}, v^{\prime\prime\times M}, d)$, followed by
the previously-defined momentum flip operator $\mathbf{F}$ which flips the
direction variable $d$, i.e.\ $\mathbf{F}\xi = (x, v, -d)$.
%
Using these, we can express a complete molecular dynamics update step as
$\mathbf{FL}_{\theta}\xi = \xip$, where now the Metropolis-Hastings acceptance
probability for this proposal is given by
%
\begin{equation}
    A(\mathbf{F}\mathbf{L} \xi | \xi) = \min\left(1,
        \frac{p(\mathbf{F}\mathbf{L}\xi)}{p(\xi)}\left|
        \frac{\partial\left[\mathbf{F}\mathbf{L}\xi\right]}
            {\partial\xi^{T}}\right|\right),
\end{equation}
%
Where $\left|\frac{\partial\left[\mathbf{F}\mathbf{L}\xi\right]}
{\partial\xi^{T}}\right|$ denotes the determinant of the Jacobian describing
the transformation.

In contrast to generic HMC where
$\left|\frac{\partial\left[\mathbf{F}\mathbf{L}\xi\right]}
{\partial\xi^{T}}\right| = 1$, we now have non-symplectic transformations
(i.e.\ non-volume preserving) and so we must explicitly account for the
determinant of the Jacobian.
%
These non-volume preserving transformations have the effect of deforming the
energy landscape, which, depending on the nature of the transformation, may
allow for the exploration of regions of space which were previously
inaccessible.
%
\newcommand{\energyA}{\includegraphics[width=0.4\textwidth]{energy_landscape/original_energy_landscape.pdf}}
\newcommand{\energyB}{\includegraphics[width=0.4\textwidth]{energy_landscape/modified_energy_landscape.pdf}}
%
\begin{figure}
  \centering
  \Huge
  \parbox{\widthof{\energyA}}{\energyA} $\overset{\mathcal{J}}{\longrightarrow}$
  \parbox{\widthof{\energyB}}{\energyB}
  \normalsize
  \caption{Example of how the determinant of the Jacobian can deform the energy landscape.}
\end{figure}

To simplify our notation, introduce an additional operator $\mathbf{R}$ that
re-samples the momentum and direction, e.g.\ given $\xi = (x, v, d)$,
$\mathbf{R}\,\xi = (x, v^{\prime}, d^{\prime})$ where $v^{\prime} \sim
\mathcal{N}(0, I)$, $d^{\prime} \sim \mathcal{U}\left(\{-1, 1\}\right)$.
%
A complete sampling step of our algorithm then consists of the following two
steps:
%
\begin{enumerate}
    \item $\xi^{\prime} = \mathbf{FL}_{\theta} \,\xi$ with probability
        $A(\mathbf{FL}_{\theta}\,\xi|\xi)$ %(Eq.~\ref{eq:metropolis_hastings}),
        otherwise $\xi^{\prime} = \xi$.
    \item $\xi^{\prime} = \mathbf{R}\,\xi$.
\end{enumerate}
%
Note however, that for MH to be well-defined, this deterministic operator must
be \emph{invertible} and \emph{have a tractable Jacobian} (i.e.\ we can compute
its determinant).
%
In order to make this operator invertible, we augment the state space $(x, v)$
into $(x, v, d)$, where $d \in \{-1, 1\}$ is drawn with equal probability and
represent the direction of the update.
%
All of the previous expressions for the augmented leapfrog updates represent
the forward ($d = 1$) direction.
%
We can derive the expressions for the backward direction ($d = -1$) by
reversing the order of the updates (i.e.\ $\vpp \rightarrow \vp$, then $\xpp
\rightarrow \xp$, followed by $\xp \rightarrow x$ and finally $\vp \rightarrow
v$).
%
For completeness, we include in Sec.~\ref{sec:lf_forward} and
Sec~\ref{sec:lf_backward} all of the equations (both forward and backward
directions) relevant for updating the variables of interest in our augmented
leapfrog sampler.
\section{Forward Direction \texorpdfstring{$(d = 1)$}{(d = 1)}:}%
\label{sec:lf_forward}
% \vspace{-20pt}
\begin{align}
  \vp &= v \odot \exp{\left(\frac{\eps}{2}S_{v}(\zeta_{1})\right)}%
        - \frac{\eps}{2}\left[\partial_{x}\,U(x)\odot \exp{\left(\eps
          Q_{v}(\zeta_{1})\right)}%
        + T_{v}(\zeta_{1})\right] \\
  \xp &= x_{\bar{m}^{t}} + m^{t}\odot \left[x \odot \exp{\left(\eps
    S_{x}(\zeta_{2})\right)}%
        + \eps\left(\vp\odot\exp{\left(\eps Q_{x}(\zeta_{2})\right)}
          + T_{x}(\zeta_{2})\right)\right] \\
  \xpp &= x^{\prime}_{m^{t}} + \bar{m}^{t}\odot \left[\xp \odot \exp{\left(\eps
    S_{x}(\zeta_{3})\right)}%
        + \eps\left(\vp\odot\exp{\left(\eps Q_{x}(\zeta_{3})\right)} +
      T_{x}(\zeta_{3})\right)\right] \\
  \vpp &= \vp \odot \exp{\left(\frac{\eps}{2}S_{v}(\zeta_{4})\right)}%
        - \frac{\eps}{2}\left[\partial_{x}\,U(\xpp)\odot \exp{\left(\eps
          Q_{v}(\zeta_{4})\right)}%
          + T_{v}(\zeta_{4})\right]
\label{eq:forward_update}
\end{align}
%
With $\zeta_{1} = (x, \partial_{x}\, U(x), t)$, $\zeta_{2} = (x_{\bar{m}^{t}},
v, t)$, $\zeta_{3} = (x^{\prime}_{m^{t}}, v, t)$, $\zeta_{4} = (\xpp,
\partial_{x}\, U(\xpp), t)$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\section{Backward Direction \texorpdfstring{$(d = -1)$}{(d = -1)}:}%
\label{sec:lf_backward}
%
\begin{align}
  v^{\prime} &= {\left\{v + \frac{\eps}{2}\left[\partial_{x}\,U(x)\odot
        \exp{\left(\eps Q_{v}(\zeta_{1})\right)}%
    + T_{v}(\zeta_{1})\right]\right\}} \odot
    \exp{\left(-\frac{\eps}{2}S_{v}(\zeta_{1})\right)} \\
  \xp &= x_{m^{t}} + \bar{m}^{t}\odot%
    {\left[x - \eps{\left(\exp{\left(\eps Q_{x}(\zeta_{2})\right)}\odot \vp%
    + T_{x}(\zeta_{2})\right)}\right]}\odot \exp{\left(-\eps
    S_{x}(\zeta_{2})\right)} \\
  \xpp &= x_{\bar{m}^{t}} + m^{t}\odot%
    {\left[\xp - \eps{\left(\exp{\left(\eps Q_{x}(\zeta_{3})\right)}\odot \vp%
    + T_{x}(\zeta_{3})\right)}\right]}\odot \exp{\left(-\eps
    S_{x}(\zeta_{3})\right)} \\
  v^{\prime\prime} &= {\left\{\vp +
      \frac{\eps}{2}\left[\partial_{x}\,U(\xpp)\odot%
        \exp{\left(\eps Q_{v}(\zeta_{1})\right)}
  + T_{v}(\zeta_{1})\right]\right\}}\odot
    \exp{\left(-\frac{\eps}{2}S_{v}(\zeta_{4})\right)}
\label{eq:backward_update}
\end{align}
%
With $\zeta_{1} = (x, \partial_{x}\, U(x), t)$, $\zeta_{2} = (x_{m^{t}}, v,
t)$, $\zeta_{3} = (x^{\prime}_{\bar{m}^{t}}, v, t)$, $\zeta_{4} = (\xpp,
\partial_{x}\, U(\xpp), t)$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Determinant of the Jacobian}
In terms of the auxiliary functions $S_{i}, Q_{i}, T_{i}$, we can compute the
Jacobian:
%
\begin{align}
  \log|\mathcal{J}|
  &= \log\bigg|\frac{\partial{\left[\mathbf{FL}_{\theta}\xi\right]}}{\partial
  \xi^{T}}\bigg|\\
  &= d \sum_{t\leq N_{\mathrm{LF}}}
    {\left[\frac{\eps}{2} \mathbbm{1}\cdot S_{v}(\zeta_{1}^{t}) + \eps m^{t}
        \cdot S_{x}(\zeta_{2}^{t})
      + \eps \bar{m}^{t} \cdot S_{x}(\zeta_{3}^{t}) + \frac{\eps}{2}\mathbbm{1}
\cdot S_{v}(\zeta_{4}^{t})\right]}.  \end{align}
%
where $N_{\mathrm{LF}}$ is the number of leapfrog steps, and $\zeta_{i}^{t}$
denotes the intermediary variable $\zeta_{i}^{t}$ at time step $t$ and $d$ is
the direction of $\xi$, i.e.\ $d = 1 \,\, (-1)$ for the forward (backward)
update.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\end{document}