text:
\section{Introduction}
Observable data in nature has some parameters which are known, such as local coordinates, but also some unknown parameters such as how the data is related to other examples. Generative models, which learn a distribution over observables, are central to our understanding of patterns in nature and allow for efficient query of new unseen examples. Recently, deep generative models have received interest due to their ability to capture a broad set of features when modelling data distributions. As such, they offer direct applications such as synthesising high fidelity images \citep{karras2020analyzing}, super-resolution \citep{dai2019second}, speech synthesis \citep{li2019neural}, and drug discovery \citep{segler2018planning}, as well as benefits for downstream tasks like semi-supervised learning \citep{chen2020generative}

\begin{figure}[b!]
\vspace*{-0.8em}
\centering
\begin{subfigure}{0.24\textwidth}
    \centering
    \begin{tikzpicture}[shorten >=1pt,->]
        \tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
        \tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
        \tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
        \tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

        \node[cir,fill=black!12]  (VAE-x)    at (0,1) {$\mathbf{x}$};
        \node[dia]                (VAE-mu)   at (1,1.75) {$\boldsymbol\mu$};
        \node[dia]                (VAE-sig)  at (1,1) {$\boldsymbol\sigma$};
        \node[cir,fill=black!12]  (VAE-eps)  at (1,0.25) {$\boldsymbol\epsilon$};
        \node[dia,fill=black!12]  (VAE-z)    at (2,1) {$\mathbf{z}$};
        \node[cir]                (VAE-xh)   at (3,1) {$\mathbf{\hat{x}}$};

        \draw(VAE-x) -- (VAE-mu);
        \draw(VAE-x) -- (VAE-sig);
        \draw(VAE-mu) -- (VAE-z);
        \draw(VAE-sig) -- (VAE-z);
        \draw(VAE-eps) -- (VAE-z);
        \draw(VAE-z) -- (VAE-xh);

        \node[align=center] at (0.45,0.7) {$E$};
        \node[align=center] at (2.45,1.3) {$D$};
        \node[align=center] at (2.3,0.45) {$\boldsymbol\mu + \boldsymbol\sigma \odot  \boldsymbol\epsilon $};
    \end{tikzpicture}
    \caption{VAE}
    \label{fig:vae}
\end{subfigure}
\hspace{\fill}
\begin{subfigure}{0.24\textwidth}
    \centering
    \vspace*{1.3em}
    \begin{tikzpicture}[shorten >=1pt,->]
        \tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
        \tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
        \tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
        \tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

        \draw[rounded corners=0pt,->] (0,1.5) -- (0,1.0) -- (0.7,1.0);

        \node[dia,fill=black!12]                (GON-0)    at (0,1.5) {$\mathbf{0}$};
        \node[cir]                (GON-2)    at (1,1.0) {$\mathbf{\hat{x}}$};

        \draw[densely dashed] (GON-2.north) to [out=90,in=0] (GON-0.east);
        \node[align=center] at (0.42,1.2) {$F$};

    \end{tikzpicture}
    \vspace*{1.5em}
    \caption{GON}
    \label{fig:gon}
\end{subfigure}
\hspace{\fill}
\begin{subfigure}{0.24\textwidth}
    \centering
    \vspace*{1.3em}
    \begin{tikzpicture}[shorten >=1pt,->]
        \tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
        \tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
        \tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
        \tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

        \draw (0,1) -- (0,0);

        \node[dia,fill=black!12]                (GON-0)    at (0,1) {$\mathbf{0}$};
        \node[cir,fill=black!12]  (GON-1)    at (0,0) {$\mathbf{c}$};
        \node[cir]                (GON-2)    at (1,0.5) {$\mathbf{\hat{x}}$};
        \draw (0,0.5) -> (GON-2);
        \draw[densely dashed] (GON-2.north) to [out=90,in=0] (GON-0.east);
        \node[align=center] at (0.42,0.7) {$F$};

    \end{tikzpicture}
    \caption{Implicit GON}
    \label{fig:implicit-gon}
\end{subfigure}
\hspace{\fill}
\begin{subfigure}{0.24\textwidth}
    \centering
    \begin{tikzpicture}[shorten >=1pt,->]
        \tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
        \tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
        \tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
        \tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

        \node[dia,fill=black!12]  (VAE-0)    at (2,1.75) {$\mathbf{0}$};
        \node[dia]                (VAE-mu)   at (1,1.75) {$\boldsymbol\mu$};
        \node[dia]                (VAE-sig)  at (1,1) {$\boldsymbol\sigma$};
        \node[cir,fill=black!12]  (VAE-eps)  at (1,0.25) {$\boldsymbol\epsilon$};
        \node[dia,fill=black!12]  (VAE-z)    at (2,1) {$\mathbf{z}$};
        \node[cir]                (VAE-xh)   at (3,1) {$\mathbf{\hat{x}}$};

        \draw(VAE-0) -- (VAE-mu);
        \draw(VAE-0) -- (VAE-sig);
        \draw(VAE-mu) -- (VAE-z);
        \draw(VAE-sig) -- (VAE-z);
        \draw(VAE-eps) -- (VAE-z);
        \draw(VAE-z) -- (VAE-xh);

        \draw[densely dashed] (VAE-xh.north) to [out=90,in=0] (VAE-0.east);

        \node[align=center] at (2.45,1.3) {$F$};
        \node[align=center] at (2.3,0.45) {$\boldsymbol\mu + \boldsymbol\sigma \odot  \boldsymbol\epsilon $};
    \end{tikzpicture}
    \caption{Variational GON}
    \label{fig:vae-gon}
\end{subfigure}
\caption{Gradient Origin Networks (GONs; b) use gradients (dashed lines) as encodings thus only a single network $F$ is required, which can be an implicit representation network (c). Unlike VAEs (a) which use two networks, $E$ and $D$, variational GONs (d) permit sampling with only one network.}
\label{fig:network-diagrams}
\end{figure}

A number of methods have been proposed such as Variational Autoencoders (VAEs, Figure~\ref{fig:vae}), which learn to encode the data to a latent space that follows a normal distribution permitting sampling \citep{Kingma2014AutoEncodingVariationalBayes}. Generative Adversarial Networks (GANs) have two competing networks, one which generates data and another which discriminates from implausible results \citep{Goodfellow2014GenerativeAdversarialNets}. Variational approaches that approximate the posterior using gradient descent \citep{lipton2017precise} and short run MCMC \citep{Nijkamp2020LearningMultilayerLatent} respectively have been proposed, but to obtain a latent vector for a sample, they require iterative gradient updates. Autoregressive Models \citep{VanDenOord2016PixelRecurrentNeural} decompose the data distribution as the product of conditional distributions and Normalizing Flows \citep{Rezende2015VariationalInferenceNormalizing} chain together invertible functions; both methods allow exact likelihood inference. Energy-Based Models (EBMs) map data points to energy values proportional to likelihood thereby permitting sampling through the use of Monte Carlo Markov Chains \citep{Du2019ImplicitGenerationGeneralization}. In general to support encoding, these approaches require separate encoding networks, are limited to invertible functions, or require multiple sampling steps.

Implicit representation learning \citep{Park2019DeepSDFLearningContinuous,Tancik2020FourierFeaturesLet}, where a network is trained on data parameterised continuously rather than in discrete grid form, has seen a surge of interest due to the small number of parameters, speed of convergence, and ability to model fine details. In particular, sinusoidal representation networks (SIRENs) \citep{Sitzmann2020ImplicitNeuralRepresentationsa} achieve impressive results, modelling many signals with high precision, thanks to their use of periodic activations paired with carefully initialised MLPs. So far, however, these models have been limited to modelling single data samples, or use an additional hypernetwork or meta learning \citep{Sitzmann2020MetaSDFMetalearningSigned} to estimate the weights of a simple implicit model, adding significant complexity.

This paper proposes Gradient Origin Networks (GONs), a new type of generative model (Figure~\ref{fig:gon}) that do not require encoders or hypernetworks. This is achieved by initialising latent points at the origin, then using the gradient of the log-likelihood of the data with respect to these points as the latent space. At inference, latent vectors can be obtained in a single step without requiring iteration. GONs are shown to have similar characteristics to convolutional autoencoders and variational autoencoders using approximately half the parameters, and can be applied to implicit representation networks (such as SIRENs) allowing a space of implicit functions to be learned with a simpler overall architecture.

\section{Preliminaries}

We first introduce some background context that will be used to derive our proposed approach.

\subsection{Empirical Bayes}

\newcommand\negpad{-0.501px}

The concept of empirical Bayes \citep{robbins1956empirical,saremi2019neural}, for a random variable $\mathbf{z} \sim p_\mathbf{z}$ and particular observation $\mathbf{z}_0 \sim p_{\mathbf{z}_0}$, provides an estimator of $\mathbf{z}$ expressed purely in terms of $p({\mathbf{z}_0})$ that minimises the expected squared error. This estimator can be written as a conditional mean:
\vspace*{-6px}
\begin{equation}
    \hat{\mathbf{z}}(\mathbf{z}_0) = \int \mathbf{z} p(\mathbf{z}|\mathbf{z}_0) d\mathbf{z} = \int \mathbf{z} \frac{p(\mathbf{z},\mathbf{z}_0)}{p(\mathbf{z}_0)} d\mathbf{z}. \vspace*{\negpad}
    \label{eqn:bayes-estimator}
\end{equation}
Of particular relevance is the case where $\mathbf{z}_0$ is a noisy observation of $\mathbf{z}$ with covariance $\bm{\Sigma}$. In this case $p(\mathbf{z}_0)$ can be represented by marginalising out $\mathbf{z}$:
\vspace*{\negpad}
\begin{equation}
    p(\mathbf{z}_0) = \int \frac{1}{(2\pi)^{d/2}|\det(\bm{\Sigma})|^{1/2}}\exp \Big( -(\mathbf{z}_0-\mathbf{z})^T \bm{\Sigma}^{-1}(\mathbf{z}_0-\mathbf{z})/2 \Big) p(\mathbf{z}) d\mathbf{z}. \vspace*{\negpad}
\end{equation}
Differentiating this with respect to $\mathbf{z}_0$ and multiplying both sides by $\bm{\Sigma}$ gives:
\vspace*{\negpad}
\begin{equation}
    \bm{\Sigma} \nabla_{\mathbf{z}_0} p(\mathbf{z}_0) = \int (\mathbf{z} - \mathbf{z}_0) p(\mathbf{z},\mathbf{z}_0) d\mathbf{z} = \int \mathbf{z} p(\mathbf{z},\mathbf{z}_0) d\mathbf{z} - \mathbf{z}_0p(\mathbf{z}_0). \vspace*{\negpad}
\end{equation}
After dividing through by $p(\mathbf{z}_0)$ and combining with Equation~\ref{eqn:bayes-estimator} we obtain a closed form estimator of $\mathbf{z}$ \citep{miyasawa1961empirical} written in terms of the score function $\nabla \log p(\mathbf{z}_0)$ \citep{hyvarinen2005estimation}:
\vspace*{\negpad}
\begin{equation}
    \hat{\mathbf{z}}(\mathbf{z}_0) = \mathbf{z}_0 + \bm{\Sigma} \nabla_{\mathbf{z}_0} \log p(\mathbf{z}_0). \vspace*{\negpad}
    \label{eqn:gaussian-empirical-bayes}
\end{equation}
This optimal procedure is achieved in what can be interpreted as a single gradient descent step, with no knowledge of the prior $p(\mathbf{z})$. By rearranging Equation~\ref{eqn:gaussian-empirical-bayes}, a definition of $\nabla \log p(\mathbf{z}_0)$ can be derived; this can be used to train models that approximate the score function \citep{song2019generative}.

\subsection{Variational Autoencoders}
Variational Autoencoders (VAEs; \citealt{Kingma2014AutoEncodingVariationalBayes}) are a probabilistic take on standard autoencoders that permit sampling. A latent-based generative model $p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})$ is defined with a normally distributed prior over the latent variables, $p_{\boldsymbol{\theta}}(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\bm{I}_d)$. $p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})$ is typically parameterised as a Bernoulli, Gaussian, multinomial distribution, or mixture of logits. In this case, the true posterior $p_{\boldsymbol{\theta}}(\mathbf{z}|\mathbf{x})$ is intractable, so a secondary encoding network $q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})$ is used to approximate the true posterior; the pair of networks thus resembles a traditional autoencoder. This allows VAEs to approximate $p_{\boldsymbol{\theta}}(\mathbf{x})$ by maximising the evidence lower bound (ELBO), defined as:
\vspace*{\negpad}
\begin{equation}
    \log p_{\boldsymbol{\theta}}(\mathbf{x}) \geq \mathcal{L}^{\textup{VAE}} = - D_{\textup{KL}}(\mathcal{N}(q_{\boldsymbol\phi}(\mathbf{z}|\mathbf{x})) || \mathcal{N}(\mathbf{0}, \bm{I}_d)) + \mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}[ \log p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}) ].
\end{equation}
To optimise this lower bound with respect to ${\boldsymbol{\theta}}$ and ${\boldsymbol{\phi}}$, gradients must be backpropagated through the stochastic process of generating samples from $\mathbf{z}' \sim q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})$. This is permitted by reparameterising $\mathbf{z}$ using the differentiable function $\mathbf{z}'= \mu(\mathbf{z}) + \sigma(\mathbf{z}) \odot \boldsymbol{\epsilon}$, where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \bm{I}_d)$ and $\mu(\mathbf{z})$ and $\sigma(\mathbf{z})^2$ are the mean and variance respectively of a multivariate Gaussian distribution with diagonal covariance.

\section{Method}

Consider some dataset $\mathbf{x} \sim p_d$ of continuous or discrete signals $\mathbf{x} \in \reals^m$, it is typical to assume that the data can be represented by low dimensional latent variables $\mathbf{z} \in \reals^k$, which can be used by a generative neural network to reconstruct the data. These variables are often estimated through the use of a secondary encoding network that is trained concurrently with the generative network. An encoding network adds additional complexity (and parameters) to the model, it can be difficult to balance capacities of the two networks, and for complex hierarchical generative models designing a suitable architecture can be difficult. This has led some to instead approximate latent variables by performing gradient descent on the generative network \citep{bojanowski2018optimizing,Nijkamp2020LearningMultilayerLatent}. While this addresses the aforementioned problems, it significantly increases the run time of the inference process, introduces additional hyperparameters to tune, and convergence is not guaranteed.

\subsection{Gradient Origin Networks \label{sec:gons}}

We propose a generative model that consists only of a decoding network, using empirical Bayes to approximate the posterior in a single step. That is, for some data point $\mathbf{x}$ and latent variable $\mathbf{z} \sim p_\mathbf{z}$, we wish to find an approximation of $p(\mathbf{z}|\mathbf{x})$. Given some noisy observation $\mathbf{z}_0=\mathbf{z}+\mathcal{N}(\mathbf{0},\bm{I}_d)$ of $\mathbf{z}$ then empirical Bayes can be applied to approximate $\mathbf{z}$. Specifically, since we wish to approximate $\mathbf{z}$ conditioned on $\mathbf{x}$, we instead calculate $\hat{\mathbf{z}}_\mathbf{x}$, the least squares estimate of $p(\mathbf{z}|\mathbf{x})$ (proof in Appendix~\ref{apx:empirical-bayes-proof}):
\begin{equation}
    \hat{\mathbf{z}}_\mathbf{x}(\mathbf{z}_0) = \mathbf{z}_0 + \nabla_{\mathbf{z}_0} \log p(\mathbf{z}_0|\mathbf{x}).
    \label{eqn:gon-bayes-simple}
\end{equation}
Using Bayes' rule, $\log p(\mathbf{z}_0|\mathbf{x})$ can be written as $\log p(\mathbf{z}_0|\mathbf{x}) = \log p(\mathbf{x}|\mathbf{z}_0) + \log p(\mathbf{z}_0) - \log p(\mathbf{x})$. Since $\log p(\mathbf{x})$ is a normalising constant that does not affect the gradient, we can rewrite Equation~\ref{eqn:gon-bayes-simple} in terms only of the decoding network and $p(\mathbf{z}_0)$:
\begin{equation}
    \hat{\mathbf{z}}_\mathbf{x}(\mathbf{z}_0) = \mathbf{z}_0 + \nabla_{\mathbf{z}_0} \Big( \log p(\mathbf{x}|\mathbf{z}_0) + \log p(\mathbf{z}_0) \Big).
    \label{eqn:gon-bayes-equation}
\end{equation}
It still remains, however, how to construct a noisy estimate of $\mathbf{z}_0$ with no knowledge of $\mathbf{z}$. If we assume $\mathbf{z}$ follows a known distribution, then it is possible to develop reasonable estimates. For instance, if we assume $p(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\bm{I}_d)$ then we could sample from $p(\mathbf{z}_0)=\mathcal{N}(\mathbf{z}_0;\mathbf{0},2\bm{I}_d)$ however this could be far from the true distribution of $p(\mathbf{z}_0|\mathbf{z})=\mathcal{N}(\mathbf{z}_0;\mathbf{z},\bm{I}_d)$. Instead we propose initialising $\mathbf{z}_0$ at the origin since this is the distribution's mean. Initialising at a constant position decreases the input variation and thus simplifies the optimisation procedure. Naturally, how $p(\mathbf{x}|\mathbf{z})$ is modelled affects $\hat{\mathbf{z}}_{\mathbf{x}}$. While mean-field models result in $\hat{\mathbf{z}}_{\mathbf{x}}$ that are linear functions of $\mathbf{x}$, conditional autoregressive models, for instance, result in non-linear $\hat{\mathbf{z}}_{\mathbf{x}}$; multiple gradient steps also induce non-linearity, however, we show that a single step works well on high dimensional data suggesting that linear encoders, which normally do not scale to high dimensional data are effective in this case.

\subsection{Autoencoding with GONs}

Before exploring GONs as generative models, we discuss the case where the prior $p(\mathbf{z})$ is unknown; such a model is referred to as an autoencoder. As such, the distribution $p(\mathbf{z}_0|\mathbf{z})$ is also unknown thus it is again unclear how we can construct a noisy estimate of $\mathbf{z}$. By training a model end-to-end where $\mathbf{z}_0$ is chosen as the origin, however, a prior is implicitly learned over $\mathbf{z}$ such that it is reachable from $\mathbf{z}_0$. Although $p(\mathbf{z})$ is unknown, we do not wish to impose a prior on $\mathbf{z}_0$; the term which enforces this is in Equation~\ref{eqn:gon-bayes-equation} is $\log p(\mathbf{z}_0)$, so we can safely ignore this term and simply maximise the likelihood of the data given $\mathbf{z}_0$. Our estimator of $\mathbf{z}$ can therefore be defined simply as $\hat{\mathbf{z}}_\mathbf{x}(\mathbf{z}_0) = \mathbf{z}_0 + \nabla_{\mathbf{z}_0} \log p(\mathbf{x}|\mathbf{z}_0)$, which can otherwise be interpreted as a single gradient descent step on the conditional log-likelihood of the data. From this estimate, the data can be reconstructed by passing $\hat{\mathbf{z}}_\mathbf{x}$ through the decoder to parameterise $p(\mathbf{x}|\hat{\mathbf{z}}_\mathbf{x})$. This procedure can be viewed more explicitly when using a neural network $F \colon \reals^k \to \reals^m$ to output the mean of $p(\mathbf{x}|\hat{\mathbf{z}}_\mathbf{x})$ parameterised by a normal distribution; in this case the loss function is defined in terms of mean squared error loss $\mathcal{L}^{\text{MSE}}$:
\begin{equation}
    G_{\mathbf{x}} = \mathcal{L}^{\text{MSE}} ( \mathbf{x}, F(-\nabla_{\mathbf{z}_0} \mathcal{L}^{\text{MSE}}(\mathbf{x}, F(\mathbf{z}_0))) ).
    \label{eqn:gon}
\end{equation}
The gradient computation thereby plays a similar role to an encoder, while $F$ can be viewed as a decoder, with the outer loss term determining the overall reconstruction quality. Using a single network to perform both roles has the advantage of simplifying the overall architecture, avoiding the need to balance networks, and avoiding bottlenecks; this is demonstrated in Figure~\ref{fig:gon} which provides a visualisation of the GON process.

\subsection{Variational GONs}

The variational approach can be naturally applied to GONs, allowing sampling in a single step while only requiring the generative network, reducing the parameters necessary. Similar to before, a feedforward neural network $F$ parameterises $p(\mathbf{x}|\mathbf{z})$, while the expectation of $p(\mathbf{z}|\mathbf{x})$ is calculated with empirical Bayes. A normal prior is assumed over $p(\mathbf{z})$ thus Equation~\ref{eqn:gon-bayes-equation} can be written as:
\begin{equation}
    \hat{\mathbf{z}}_\mathbf{x}(\mathbf{z}_0) = \mathbf{z}_0 + \nabla_{\mathbf{z}_0} \Big( \log p(\mathbf{x}|\mathbf{z}_0) + \log \mathcal{N}(\mathbf{z}_0;\mathbf{0},2\bm{I}_d) \Big),
\end{equation}
where $\mathbf{z}_0 = \mathbf{0}$ as discussed in Section~\ref{sec:gons}. While it would be possible to use this estimate directly within a constant-variance VAE \citep{ghosh2019variational}, we opt to incorporate the reparameterisation trick into the generative network as a stochastic layer, to represent the distribution over which $\mathbf{x}$ could be encoded to, using empirical Bayes to estimate $\mathbf{z}$. Similar to the autoencoding approach, we could ignore $\log p(\mathbf{z}_0)$, however we find assuming a normally distributed $\mathbf{z}_0$ implicitly constrains $\mathbf{z}$, aiding the optimisation procedure. Specifically, the forward pass of $F$ is implemented as follows: $\hat{\mathbf{z}}_\mathbf{x}$ (or $\mathbf{z}_0$) is mapped by linear transformations to $\mu(\hat{\mathbf{z}}_\mathbf{x})$ and $\sigma(\hat{\mathbf{z}}_\mathbf{x})$ and the reparameterisation trick is applied, subsequently the further transformations formerly defined as $F$ in the GON formulation are applied, providing parameters for for $p(\mathbf{x}|\hat{\mathbf{z}}_\mathbf{x})$. Training is performed end-to-end, minimising the ELBO:
\begin{equation}
    \mathcal{L}^{\textup{VAE}} = - D_{\textup{KL}}(\mathcal{N}(\mu(\hat{\mathbf{z}}_\mathbf{x}), \sigma(\hat{\mathbf{z}}_\mathbf{x})^2) || \mathcal{N}(\mathbf{0}, \bm{I}_d)) + \log p(\mathbf{x}|\hat{\mathbf{z}}_\mathbf{x}) .
\end{equation}
These steps are shown in Figure~\ref{fig:vae-gon}, which in practice has a simple implementation

\subsection{Implicit GONs}

In the field of implicit representation networks, the aim is to learn a neural approximation of a function $\Phi$ that satisfies an implicit equation of the form:
\begin{equation}
    R(\mathbf{c}, \Phi, \nabla_\Phi, \nabla^2_\Phi, \dots) = 0, \quad \Phi \colon \mathbf{c} \mapsto \Phi(\mathbf{c}),
\end{equation}
where $R$'s definition is problem dependent but often corresponds to a loss function. Equations with this structure arise in a myriad of fields, namely 3D modelling, image, video, and audio representation \citep{Sitzmann2020ImplicitNeuralRepresentationsa}. In these cases, data samples $\mathbf{x} = \{(\mathbf{c}, \Phi_{\mathbf{x}}(\mathbf{c}))\}$ can be represented in this form in terms of coordinates $\mathbf{c} \in \reals^n$ and the corresponding data at those points $\Phi \colon \reals^n \to \reals^m$. Due to the continuous nature of $\Phi$, data with large spatial dimensions can be represented much more efficiently than approaches using convolutions, for instance. Despite these benefits, representing a distribution of data points using implicit methods is more challenging, leading to the use of hypernetworks which estimate the weights of an implicit network for each data point \citep{Sitzmann2020MetaSDFMetalearningSigned}; this increases the number of parameters and adds significant complexity.

By applying the GON procedure to implicit representation networks, it is possible learn a space of implicit functions without the need for additional networks. We assume there exist latent vectors $\mathbf{z} \in \reals^k$ corresponding to data samples; the concatentation of these latent variables with the data coordinates can therefore geometrically be seen as points on a manifold that describe the full dataset in keeping with the manifold hypothesis \citep{fefferman2016testing}. An implicit Gradient Origin Network can be trained on this space to mimic $\Phi$ using a neural network $F \colon \reals^{n+k} \to \reals^m$, thereby learning a space of implicit functions by modifying Equation~\ref{eqn:gon}:
\begin{equation}
    I_{\mathbf{x}} = \int \mathcal{L} \Big( \Phi_{\mathbf{x}}(\mathbf{c}), F\Big(\mathbf{c} \oplus -\nabla_{\mathbf{z}_0} \int \mathcal{L} \big( \Phi_{\mathbf{x}}(\mathbf{c}), F(\mathbf{c} \oplus \mathbf{z}_0) \big) d\mathbf{c} \Big) \Big) d\mathbf{c},
    \label{eqn:implicit-gon-loss}
\end{equation}
where both integrations are performed over the space of coordinates and $\mathbf{c} \oplus \mathbf{z}$ represents a concatenation (Figure~\ref{fig:implicit-gon}).
Similar to the non-implicit approach, the computation of latent vectors can be expressed as $\mathbf{z} = -\nabla_{\mathbf{z}_0} \int \mathcal{L} \big( \Phi_{\mathbf{x}}(\mathbf{c}), F(\mathbf{c} \oplus \mathbf{z}_0) \big) d\mathbf{c}$. In particular, we parameterise $F$ with a SIREN \citep{Sitzmann2020ImplicitNeuralRepresentationsa}, finding that it is capable of modelling both high and low frequency components in high dimensional spaces.

\subsection{GON Generalisations}

There are a number of interesting generalisations that make this approach applicable to other tasks. In Equations~\ref{eqn:gon} and \ref{eqn:implicit-gon-loss} we use the same $\mathcal{L}$ in both the inner term and outer term, however, as with variational GONs, it is possible to use different functions; through this, training a GON concurrently as a generative model and classifier is possible, or through some minor modifications to the loss involving the addition of the categorical cross entropy loss function $\mathcal{L}^{\textsc{CCE}}$ to maximise the likelihood of classification, solely as a classifier:
\begin{equation}
    C_{\mathbf{x}} = \mathcal{L}^{\textsc{CCE}}(f( -\nabla_{\mathbf{z}_0} \mathcal{L} ( \mathbf{x}, F(\mathbf{z}_0) ) ), \mathbf{y}),
    \label{eqn:gon-classify}
\end{equation}
where $\mathbf{y}$ are the target labels and $f$ is a single linear layer followed by softmax. Another possibility is modality conversion for translation tasks; in this case the inner reconstruction loss is performed on the source signal and the outer loss on the target signal.

\subsection{Justification}

Beyond empirical Bayes, we provide some additional analysis on why a single gradient step is in general sufficient as an encoder. Firstly, the gradient of the loss inherently offers substantial information about data making it a good encoder. Secondly, a good latent space should satisfy local consistency \citep{zhou2003learning, kamnitsas2018semi}. GONs satisfy this since similar data points will have similar gradients due to the constant latent initialisation. As such, the network needs only to find an equilibrium where its prior is the gradient operation, allowing for significant freedom. Finally, since GONs are trained by backpropagating through empirical Bayes, there are benefits to using an activation function whose second derivative is non-zero.

\section{Results}
We evaluate Gradient Origin Networks on a variety of image datasets: MNIST \citep{lecun1998gradient}, Fashion-MNIST \citep{xiao2017fashion}, Small NORB \citep{lecun2004learning}, COIL-20 \citep{nane1996columbia}, CIFAR-10 \citep{krizhevsky2009learning}, CelebA \citep{liu2015faceattributes}, and LSUN Bedroom \citep{yu2015lsun}. Simple models are used: for small images, implicit GONs consist of approximately 4 hidden layers of 256 units and convolutional GONs consist of 4 convolution layers with Batch Normalization \citep{ioffe2015batch} and the ELU non-linearity \citep{clevert2015fast}, for larger images the same general architecture is used, scaled up with additional layers; all training is performed with the Adam optimiser \citep{Kingma2017AdamMethodStochastic}. Error bars in graphs represent standard deviations over three runs.

\begin{table}[H]
\centering
\begin{tabular}{lccccc}
                            & MNIST       & Fashion-MNIST & Small NORB  & COIL20      & CIFAR-10     \\ \hline
Single Step \\ \hline
GON (ours)                  & $\bf{0.41\!\pm\!0.01}$    & $\bf{1.00\!\pm\!0.01}$      & $\bf{0.17\!\pm\!0.01}$    & $5.35\!\pm\!0.01$        & $\bf{9.12\!\pm\!0.03}$         \\
AE                          & $1.33\!\pm\!0.02$        & $1.75\!\pm\!0.02$          & $0.73\!\pm\!0.01$        & $6.05\!\pm\!0.11$        & $12.24\!\pm\!0.05$        \\
Tied AE                  & $2.16\!\pm\!0.03$        & $2.45\!\pm\!0.02$          & $0.93\!\pm\!0.03$        & $6.68\!\pm\!0.13$        & $14.12\!\pm\!0.34$        \\
1 Step Detach             & $8.17\!\pm\!0.15$        & $7.76\!\pm\!0.21$          & $1.84\!\pm\!0.01$        & $15.72\!\pm\!0.70$       & $30.17\!\pm\!0.29$        \\
\hline
Multiple Steps \\ \hline
10 Step Detach           & $8.13\!\pm\!0.50$        & $7.22\!\pm\!0.92$          & $1.78\!\pm\!0.02$        & $15.48\!\pm\!0.60$       & $28.68\!\pm\!1.16$        \\
10 Step                    & $0.42\!\pm\!0.01$        & $1.01\!\pm\!0.01$      & $\bf{0.17\!\pm\!0.01}$    & $5.36\!\pm\!0.01$        & $9.19\!\pm\!0.01$         \\
GLO                         & $0.70\!\pm\!0.01$        & $1.78\!\pm\!0.01$          & $0.61\!\pm\!0.01$        & $\bf{3.27\!\pm\!0.02}$    & $\bf{9.12\!\pm\!0.03}$  \\
\end{tabular}
\caption{Validation reconstruction loss (summed squared error) over 500 epochs. For GLO, latents are assigned to data points and jointly optimised with the network. GONs significantly outperform other single step methods and achieve the lowest reconstruction error on four of the five datasets.}
\label{tab:gon-recons}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{lcccccc}
             & MNIST     & Fashion-MNIST & Small NORB     & COIL20     & CIFAR-10 & CelebA   \\ \hline
VGON (ours)  & \bf 1.06  & 3.30          & \bf 2.34       & \bf 3.44   & \bf 5.85 & \bf 5.41 \\
VAE          & 1.15      & \bf 3.23      & 2.54           & 3.63       & 5.94     & 5.59     \\

\end{tabular}
\caption{Validation ELBO in bits/dim over 1000 epochs (CelebA is trained over 150 epochs).}
\label{tab:gon-vae}
\end{table}

\subsection{Quantitative Evaluation}

A quantitative evaluation of the representation ability of GONs is performed in Table~\ref{tab:gon-recons} against a number of baseline approaches. We compare against the single step methods: standard autoencoder, an autoencoder with tied weights \citep{gedeon1998stochastic}, and a GON with gradients detached from $\mathbf{z}$, as well as the multi-step methods: 10 gradient descent steps per data point, with and without detaching gradients, and a GLO \citep{bojanowski2018optimizing} which assigns a persistent latent vector to each data point and optimises them with gradient descent, therefore taking orders of magnitude longer to reconstruct validation data than other approaches. For the 10-step methods, a learning rate of 0.1 is applied as used in other literature \citep{nijkamp2019learning}; the GLO is trained with MSE for consistency with the other approaches and we do not project latents onto a hypersphere as proposed by the authors since in this experiment sampling is unimportant and this would handicap the approach. GONs achieve much lower validation loss than other single step methods and are competitive with the multi-step approaches; in fact, GONs achieve the lowest loss on four out of the five datasets.

\begin{figure}[b!]

\begin{subfigure}{0.32\textwidth}
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Validation Loss},
    xtick pos=left,
    ytick pos=left,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,
    legend style={draw=none},
    legend columns = 2,

]
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=5] table [x=Epoch, y=AETest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};\addlegendentry{AE 1x1}
\addplot+[spectral3, mark options={fill=spectral3}, mark repeat=5] table [x=Epoch, y=GONTest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};\addlegendentry{GON 1x1}

\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=5] table [x=Epoch, y=AETest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};\addlegendentry{AE 2x2}
\addplot+[spectral4, mark options={fill=spectral4}, mark repeat=5] table [x=Epoch, y=GONTest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};\addlegendentry{GON 2x2}

\addplot+[spectral5, mark options={fill=spectral5}, mark repeat=5] table [x=Epoch, y=AETest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};\addlegendentry{AE 4x4}
\addplot+[spectral8, mark options={fill=spectral8}, mark repeat=5] table [x=Epoch, y=GONTest, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};\addlegendentry{GON 4x4}

\addplot [name path=AE1upper,draw=none] table [x=Epoch,y expr=\thisrow{AETest}+\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=AE1lower,draw=none] table [x=Epoch,y expr=\thisrow{AETest}-\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral2!20] fill between[of=AE1upper and AE1lower];

\addplot [name path=GON1upper,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}+\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=GON1lower,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}-\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-1x1-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral3!20] fill between[of=GON1upper and GON1lower];

\addplot [name path=AE2upper,draw=none] table [x=Epoch,y expr=\thisrow{AETest}+\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=AE2lower,draw=none] table [x=Epoch,y expr=\thisrow{AETest}-\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral10!20] fill between[of=AE2upper and AE2lower];

\addplot [name path=GON2upper,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}+\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=GON2lower,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}-\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-2x2-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral4!20] fill between[of=GON2upper and GON2lower];

\addplot [name path=AE4upper,draw=none] table [x=Epoch,y expr=\thisrow{AETest}+\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=AE4lower,draw=none] table [x=Epoch,y expr=\thisrow{AETest}-\thisrow{AETestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral5!20] fill between[of=AE4upper and AE4lower];

\addplot [name path=GON4upper,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}+\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};
\addplot [name path=GON4lower,draw=none] table [x=Epoch,y expr=\thisrow{GONTest}-\thisrow{GONTestSTD}, col sep=comma] {figures/gon-vs-ae/gon-vs-ae-4x4-nz=512,f=16,mean,eval,std.csv};
\addplot [fill=spectral8!20] fill between[of=GON4upper and GON4lower];

\end{axis}
\end{tikzpicture}
\end{adjustbox}
\caption{GONs achieve lower validation loss than autoencoders.}
\label{fig:gon-vs-ae-plot}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Validation Loss},
    xtick pos=left,
    ytick pos=left,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,
    legend style={draw=none},
    legend columns=2,

]
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=5] table [x=Epoch, y=1_step_test, col sep=comma] {figures/gon-multistep/multistep.csv};\addlegendentry{GON\textsuperscript{\textdagger}}

\addplot+[spectral3, mark options={fill=spectral3}, mark repeat=5] table [x=Epoch, y={1 step test}, col sep=comma] {figures/gon-multistep/detached_latents.csv};\addlegendentry{1 Step*}

\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=5] table [x=Epoch, y=2_step_test, col sep=comma] {figures/gon-multistep/multistep.csv};\addlegendentry{2 Steps\textsuperscript{\textdagger}}

\addplot+[spectral5, mark options={fill=spectral5}, mark repeat=5] table [x=Epoch, y={10 step test}, col sep=comma] {figures/gon-multistep/detached_latents.csv};\addlegendentry{10 Steps*}

\addplot+[spectral8, mark options={fill=spectral8}, mark repeat=5] table [x=Epoch, y=8_step_test, col sep=comma] {figures/gon-multistep/multistep.csv};\addlegendentry{8 Steps\textsuperscript{\textdagger}}

\addplot+[spectral11, mark options={fill=spectral11}, mark repeat=5] table [x=Epoch, y={25 steps (lr=1e-2) test}, col sep=comma] {figures/gon-multistep/detached_latents.csv};\addlegendentry{25 Steps*}
\end{axis}
\end{tikzpicture}
\end{adjustbox}
\caption{Multiple latent update steps.\\ *=grads detached, \textsuperscript{\textdagger}=not detached.}
\label{fig:gon-multiple-steps-plot}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Reconstruction Loss},
    xtick pos=left,
    ytick pos=left,
    ymax=0.007,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,
    legend style={draw=none},
    legend columns=2,

]
\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=50] table [x=Epoch, y={AETrain}, col sep=comma] {figures/long_plots/cifar_long.csv};\addlegendentry{AE Train}
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=50] table [x=Epoch, y={GON Train}, col sep=comma] {figures/long_plots/cifar_long.csv};\addlegendentry{GON Train}
\addplot+[spectral8, mark options={fill=spectral8}, mark repeat=50] table [x=Epoch, y={AE Test}, col sep=comma] {figures/long_plots/cifar_long.csv};\addlegendentry{AE Test}
\addplot+[spectral4, mark options={fill=spectral4}, mark repeat=50] table [x=Epoch, y={GON Test}, col sep=comma] {figures/long_plots/cifar_long.csv};\addlegendentry{GON Test}

\end{axis}
\end{tikzpicture}
\end{adjustbox}
\caption{GONs overfit less than standard autoencoders.}
\label{fig:gon-overfit-plot}
\end{subfigure}
\caption{Gradient Origin Networks trained on CIFAR-10 are found to outperform autoencoders using exactly the same architecture without the encoder, requiring half the number of parameters.}
\label{fig:gon-vs-ae}
\end{figure}

\begin{figure}[b!]
\centering

\begin{adjustbox}{width=0.6\textwidth}
\begin{tikzpicture}
\matrix[
        matrix of nodes,
        anchor=south,
        draw=none,
        inner sep=0.2em,
    ] {
        \ref{plots:sin}& sin&[4pt]
        \ref{plots:relu}& ReLU&[4pt]
        \ref{plots:leakyrelu}& LeakyReLU&[4pt]
        \ref{plots:tanh}& tanh&[4pt]
        \ref{plots:elu}& ELU&[4pt]
        \ref{plots:softplus}& softplus&[4pt]
        \ref{plots:swish}& swish&[4pt]\\
    };
\end{tikzpicture}
\end{adjustbox}

\begin{subfigure}{0.32\textwidth}
\centering
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Reconstruction Loss},
    xtick pos=left,
    ytick pos=left,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,

]
\addplot+[spectral1, mark options={fill=spectral1}, mark repeat=2] table [x=Epoch, y=sin, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{$\sin$}\label{plots:sin}
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=2] table [x=Epoch, y=RELU, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{ReLU}\label{plots:relu}
\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=2] table [x=Epoch, y=leakyrelu, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{LeakyReLU}\label{plots:leakyrelu}
\addplot+[spectral11, mark options={fill=spectral11}, mark repeat=2] table [x=Epoch, y=tanh, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{tanh}\label{plots:tanh}
\addplot+[spectral9, mark options={fill=spectral9}, mark repeat=2] table [x=Epoch, y=softplus, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{softplus}\label{plots:softplus}
\addplot+[spectral3, mark options={fill=spectral3}, mark repeat=2] table [x=Epoch, y=swish, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{swish}\label{plots:swish}
\addplot+[spectral5, mark options={fill=spectral5}, mark repeat=2] table [x=Epoch, y=ELU, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};\addlegendentry{ELU}\label{plots:elu}
\legend{};

\addplot [name path=sinupper,draw=none] table [x=Epoch,y expr=\thisrow{sin}+\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=sinlower,draw=none] table [x=Epoch,y expr=\thisrow{sin}-\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral1!20] fill between[of=sinupper and sinlower];

\addplot [name path=RELUupper,draw=none] table [x=Epoch,y expr=\thisrow{RELU}+\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=RELUlower,draw=none] table [x=Epoch,y expr=\thisrow{RELU}-\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral2!20] fill between[of=RELUupper and RELUlower];

\addplot [name path=leakyreluupper,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}+\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=leakyrelulower,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}-\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral10!20] fill between[of=leakyreluupper and leakyrelulower];

\addplot [name path=tanhupper,draw=none] table [x=Epoch,y expr=\thisrow{tanh}+\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=tanhlower,draw=none] table [x=Epoch,y expr=\thisrow{tanh}-\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral11!20] fill between[of=tanhupper and tanhlower];

\addplot [name path=softplusupper,draw=none] table [x=Epoch,y expr=\thisrow{softplus}+\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=softpluslower,draw=none] table [x=Epoch,y expr=\thisrow{softplus}-\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral9!20] fill between[of=softplusupper and softpluslower];

\addplot [name path=swishupper,draw=none] table [x=Epoch,y expr=\thisrow{swish}+\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=swishlower,draw=none] table [x=Epoch,y expr=\thisrow{swish}-\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral3!20] fill between[of=swishupper and swishlower];

\addplot [name path=ELUupper,draw=none] table [x=Epoch,y expr=\thisrow{ELU}+\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [name path=ELUlower,draw=none] table [x=Epoch,y expr=\thisrow{ELU}-\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=512,ngf=16,mean,std.csv};
\addplot [fill=spectral5!20] fill between[of=ELUupper and ELUlower];

\end{axis}
\coordinate (top) at (rel axis cs:0,1)
\coordinate (bot) at (rel axis cs:1,0)
\path (top|-current bounding box.north)--
      coordinate(legendpos)
      (bot|-current bounding box.north);
\end{tikzpicture}
\end{adjustbox}
\caption{512 latent variables}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Reconstruction Loss},
    xtick pos=left,
    ytick pos=left,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,

]
\addplot+[spectral1, mark options={fill=spectral1}, mark repeat=2] table [x=Epoch, y=sin, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{$\sin$}
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=2] table [x=Epoch, y=RELU, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{ReLU}
\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=2] table [x=Epoch, y=leakyrelu, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{LeakyReLU}
\addplot+[spectral11, mark options={fill=spectral11}, mark repeat=2] table [x=Epoch, y=tanh, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{tanh}
\addplot+[spectral9, mark options={fill=spectral9}, mark repeat=2] table [x=Epoch, y=softplus, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{softplus}
\addplot+[spectral3, mark options={fill=spectral3}, mark repeat=2] table [x=Epoch, y=swish, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{swish}
\addplot+[spectral5, mark options={fill=spectral5}, mark repeat=2] table [x=Epoch, y=ELU, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};\addlegendentry{ELU}
\legend{};

\addplot [name path=sinupper,draw=none] table [x=Epoch,y expr=\thisrow{sin}+\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=sinlower,draw=none] table [x=Epoch,y expr=\thisrow{sin}-\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral1!20] fill between[of=sinupper and sinlower];

\addplot [name path=RELUupper,draw=none] table [x=Epoch,y expr=\thisrow{RELU}+\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=RELUlower,draw=none] table [x=Epoch,y expr=\thisrow{RELU}-\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral2!20] fill between[of=RELUupper and RELUlower];

\addplot [name path=leakyreluupper,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}+\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=leakyrelulower,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}-\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral10!20] fill between[of=leakyreluupper and leakyrelulower];

\addplot [name path=tanhupper,draw=none] table [x=Epoch,y expr=\thisrow{tanh}+\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=tanhlower,draw=none] table [x=Epoch,y expr=\thisrow{tanh}-\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral11!20] fill between[of=tanhupper and tanhlower];

\addplot [name path=softplusupper,draw=none] table [x=Epoch,y expr=\thisrow{softplus}+\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=softpluslower,draw=none] table [x=Epoch,y expr=\thisrow{softplus}-\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral9!20] fill between[of=softplusupper and softpluslower];

\addplot [name path=swishupper,draw=none] table [x=Epoch,y expr=\thisrow{swish}+\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=swishlower,draw=none] table [x=Epoch,y expr=\thisrow{swish}-\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral3!20] fill between[of=swishupper and swishlower];

\addplot [name path=ELUupper,draw=none] table [x=Epoch,y expr=\thisrow{ELU}+\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [name path=ELUlower,draw=none] table [x=Epoch,y expr=\thisrow{ELU}-\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=128,ngf=16,mean,std.csv};
\addplot [fill=spectral5!20] fill between[of=ELUupper and ELUlower];

\end{axis}
\end{tikzpicture}
\end{adjustbox}
\caption{128 latent variables}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\begin{adjustbox}{width=\linewidth}
\begin{tikzpicture}
\begin{axis}[
    width=7cm,height=6cm,
    xlabel={Training Epoch},
    ylabel={Reconstruction Loss},
    xtick pos=left,
    ytick pos=left,
    enlarge x limits=false,
    every x tick/.style={color=black, thin},
    every y tick/.style={color=black, thin},
    tick align=outside,
    xlabel near ticks,
    ylabel near ticks,
    axis on top,

]
\addplot+[spectral1, mark options={fill=spectral1}, mark repeat=2] table [x=Epoch, y=sin, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{$\sin$}
\addplot+[spectral2, mark options={fill=spectral2}, mark repeat=2] table [x=Epoch, y=RELU, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{ReLU}
\addplot+[spectral10, mark options={fill=spectral10}, mark repeat=2] table [x=Epoch, y=leakyrelu, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{LeakyReLU}
\addplot+[spectral11, mark options={fill=spectral11}, mark repeat=2] table [x=Epoch, y=tanh, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{tanh}
\addplot+[spectral9, mark options={fill=spectral9}, mark repeat=2] table [x=Epoch, y=softplus, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{softplus}
\addplot+[spectral3, mark options={fill=spectral3}, mark repeat=2] table [x=Epoch, y=swish, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{swish}
\addplot+[spectral5, mark options={fill=spectral5}, mark repeat=2] table [x=Epoch, y=ELU, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};\addlegendentry{ELU}
\legend{};

\addplot [name path=sinupper,draw=none] table [x=Epoch,y expr=\thisrow{sin}+\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=sinlower,draw=none] table [x=Epoch,y expr=\thisrow{sin}-\thisrow{sinstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral1!20] fill between[of=sinupper and sinlower];

\addplot [name path=RELUupper,draw=none] table [x=Epoch,y expr=\thisrow{RELU}+\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=RELUlower,draw=none] table [x=Epoch,y expr=\thisrow{RELU}-\thisrow{RELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral2!20] fill between[of=RELUupper and RELUlower];

\addplot [name path=leakyreluupper,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}+\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=leakyrelulower,draw=none] table [x=Epoch,y expr=\thisrow{leakyrelu}-\thisrow{leakyrelustd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral10!20] fill between[of=leakyreluupper and leakyrelulower];

\addplot [name path=tanhupper,draw=none] table [x=Epoch,y expr=\thisrow{tanh}+\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=tanhlower,draw=none] table [x=Epoch,y expr=\thisrow{tanh}-\thisrow{tanhstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral11!20] fill between[of=tanhupper and tanhlower];

\addplot [name path=softplusupper,draw=none] table [x=Epoch,y expr=\thisrow{softplus}+\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=softpluslower,draw=none] table [x=Epoch,y expr=\thisrow{softplus}-\thisrow{softplusstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral9!20] fill between[of=softplusupper and softpluslower];

\addplot [name path=swishupper,draw=none] table [x=Epoch,y expr=\thisrow{swish}+\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=swishlower,draw=none] table [x=Epoch,y expr=\thisrow{swish}-\thisrow{swishstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral3!20] fill between[of=swishupper and swishlower];

\addplot [name path=ELUupper,draw=none] table [x=Epoch,y expr=\thisrow{ELU}+\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [name path=ELUlower,draw=none] table [x=Epoch,y expr=\thisrow{ELU}-\thisrow{ELUstd}, col sep=comma] {figures/activations/activations,cifar,nz=32,ngf=16,mean,std.csv};
\addplot [fill=spectral5!20] fill between[of=ELUupper and ELUlower];
\end{axis}
\end{tikzpicture}
\end{adjustbox}
\caption{32 latent variables}
\end{subfigure}

\caption{The impact of activation function and number of latent variables on model performance for a GON measured by comparing reconstruction losses through training.}
\label{fig:gon-activations}
\end{figure}

Our variational GON is compared with a VAE, whose decoder is the same as the GON, quantitatively in terms of ELBO on the test set in Table \ref{tab:gon-vae}. We find that the representation ability of GONs is pertinent here also, allowing the variational GON to achieve lower ELBO on five out of the six datasets tested. Both models were trained with the original VAE formulation for fairness, however, we note that variations aimed at improving VAE sample quality such as $\beta$-VAEs \citep{higgins2017beta} are also applicable to variational GONs to the same effect.

An ablation study is performed, comparing convolutional GONs with autoencoders whose decoders have exactly the same architecture as the GONs (Figure~\ref{fig:gon-vs-ae-plot}) and where the autoencoder decoders mirror their respective encoders. The mean reconstruction loss over the test set is measured after each epoch for a variety of latent sizes. Despite having half the number of parameters and linear encodings, GONs achieve significantly lower reconstruction loss over a wide range of architectures. Further experiments evaluating the impact of latent size and model capacity are performed in Appendix~\ref{apx:compare-autoencoders}.

Our hypothesis that for high dimensional datasets, a single gradient step is sufficient when jointly optimised with the forwards pass is tested on the CIFAR-10 dataset in Figure~\ref{fig:gon-multiple-steps-plot}. We observe negligible difference between a single step and multiple jointly optimised steps, in support of our hypothesis. Performing multiple steps greatly increases run-time so there is seemingly no benefit in this case. Additionally, the importance of the joint optimisation procedure is determined by detaching the gradients from $\mathbf{z}$ before reconstructing (Figure~\ref{fig:gon-multiple-steps-plot}); this results in markedly worse performance, even when in conjunction with multiple steps. In Figure~\ref{fig:gon-overfit-plot} we assess whether the greater performance of GONs relative to autoencoders comes at the expense of generalisation; we find that the opposite is true, that the discrepancy between reconstruction loss on the training and test sets is greater with autoencoders. This result extends to other datasets as can be seen in Appendix~\ref{apx:gon-generalisation}.

Figure~\ref{fig:gon-activations} demonstrates the effect activation functions have on convolutional GON performance for different numbers of latent variables. Since optimising GONs requires computing second order derivatives, the choice of nonlinearity requires different characteristics to standard models. In particular, GONs prefer functions that are not only effective activation functions, but also whose second derivatives are non-zero, unlike ReLUs where $\text{ReLU}''(x)=0$. The ELU non-linearity is effective with all tested architectures.

\begin{figure}[b!]
    \centering
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/mnist,latent=32,hidden_features=32,layers=3,params=4385,fitting.png}
        \caption{4,385 parameters}
        \label{fig:tiny-overfit}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/fashion,latent=128,hidden_features=256layers=4,params=297217,fitting.png}
        \caption{297k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/norb,latent=32,hidden_features=256layers=4,params=272449,fitting.png}
        \caption{270k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/coil,latent=32,hidden_features=256layers=4,params=272449,fitting.png}
        \caption{270k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/cifar,latent=128,hidden_features=256,layers=5,params=396000,fitting.png}
        \caption{396k parameters}
    \end{subfigure}
\caption{Training implicit GONs with few parameters demonstrates their representation ability.}
\label{fig:fitting}
\end{figure}

\begin{figure}[b!]
    \centering

    \begin{tikzpicture}
    \node (fig) {\includegraphics[width=0.99\linewidth]{figures/supersampling/coil/cherry.pdf}};
    \node (otxt) at (-6.32,0.9) {original};
    \node (gtxt) at (-4.26,0.9) {GON};
    \node (otxt2) at (1.1,0.9) {original};
    \node (gtxt2) at (3.15,0.9) {GON};
    \end{tikzpicture}

    \caption{By training an implicit GON on 32x32 images, then sampling at 256x256, super-resolution is possible despite never observing high resolution data.}
    \label{fig:supersample-coil-cherry}
\end{figure}

\subsection{Qualitative Evaluation}

The representation ability of implicit GONs is shown in Figure~\ref{fig:fitting} where we train on large image datasets using a relatively small number of parameters. In particular, Figure~\ref{fig:tiny-overfit} shows MNIST can be well-represented with just 4,385 parameters (a SIREN with 3 hidden layers each with 32 hidden units, and 32 latent dimensions).
An advantage of modelling data with implicit networks is that coordinates can be at arbitrarily high resolutions. In Figure~\ref{fig:supersample-coil-cherry} we train on 32x32 images then reconstruct at 256x256. A significant amount of high frequency detail is modelled despite only seeing low resolution images.
The structure of the implicit GON latent space is shown by sampling latent codes from pairs of images, and then spherically interpolating between them to synthesise new samples (Figure~\ref{fig:interpolations}). These samples are shown to capture variations in shape (the shoes in Figure~\ref{fig:fashion-slerp}), size, and rotation (Figure~\ref{fig:coil-slerp}).

We assess the quality of samples from variational GONs using convolutional models trained to convergence in Figure~\ref{fig:conv-samples}. These are diverse and often contain fine details. Samples from an implicit GON trained with early stopping, as a simple alternative, can be found in Appendix~\ref{apx:early-stopping} however this approach results in fine details being lost.
GONs are also found to converge quickly; we plot reconstructions at multiple time points during the first minute of training (Figure~\ref{fig:mnist-training}). After only 3 seconds of training on a single GPU, a large amount of signal information from MNIST is modelled.

\begin{figure}[b!]
    \centering
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/mnist,latent=32,hidden_features=256,layers=4,params=272449,slerp.png}
        \caption{MNIST}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/fashion,latent=512,hidden_features=256,layers=4,params=396289,slerp.png}
        \caption{Fashion-MNIST}
        \label{fig:fashion-slerp}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/norb,latent=32,hidden_features=256,layers=4,params=272449,slerp.png}
        \caption{Small NORB}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/coil20,latent=32,hidden_features=256,layers=4,params=272449,slerp.png}
        \caption{COIL20}
        \label{fig:coil-slerp}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/cifar,latent=512,hidden_features=256,layers=5,params=462595,slerp.png}
        \caption{CIFAR-10}
    \end{subfigure}
\caption{Spherical linear interpolations between points in the latent space for trained implicit GONs using different datasets (approximately 2-10 minutes training per dataset on a single GPU).}
\label{fig:interpolations}
\end{figure}

\begin{figure}[b!]
    \centering
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/vae-gon/mnist-small.png}
        \caption{4,331 parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/vae-gon/mnist.png}
        \caption{270k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/vae-gon/fashion.png}
        \caption{270k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/vae-gon/norb.png}
        \caption{270k parameters}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/vae-gon/coil.png}
        \caption{270k parameters}
    \end{subfigure}
\caption{Random samples from a convolutional variational GON with normally distributed latents.}
\label{fig:conv-samples}
\end{figure}

\begin{figure}[b!]
    \centering
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/fastmnist/g_300.jpg}
        \caption{3s 300 steps}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/fastmnist/g_1000.jpg}
        \caption{10s 1000 steps}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/fastmnist/g_3000.jpg}
        \caption{30s 3000 steps}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/fastmnist/g_6000.jpg}
        \caption{60s 6000 steps}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
       \centering
       \includegraphics[width=\linewidth]{figures/fastmnist/x.jpg}
        \caption{Ground Truth}
    \end{subfigure}
\caption{Convergence of convolutional GONs with 74k parameters.}
\label{fig:mnist-training}
\end{figure}

In order to evaluate how well GONs can represent high resolution natural images, we train a convolutional GON on the LSUN Bedroom dataset scaled to 128x128 (Figure~\ref{fig:lsun-representation}). As with smaller, more simple data, we find training to always be extremely stable and consistent over a wide range of hyperparameter settings. Reconstructions are of excellent quality given the simple network architecture.
A convolutional variational GON is also trained on the CelebA dataset scaled to 64x64 (Figure~\ref{fig:celeba-vgon}). Unconditional samples are somewhat blurry as commonly associated with traditional VAE models on complex natural images \citep{zhao2017towards} but otherwise show wide variety.

\begin{figure}[h]
    \centering
    \begin{subfigure}[t]{0.65306122449\textwidth}
        \centering
        \includegraphics[width=0.49\linewidth,trim={0 0 0 0},clip]{figures/lsun_128_groundtruth.png}
        \hfill
        \includegraphics[width=0.49\linewidth,trim={0 0 0 0},clip]{figures/lsun_128_recons.png}
        \caption{LSUN 128x128 Bedroom validation images (left) reconstructed by a convolutional GON (right).}
        \label{fig:lsun-representation}
    \end{subfigure}
    \hfill

    \begin{subfigure}[t]{0.32\textwidth}
        \centering
        \includegraphics[width=\linewidth,trim={0 0 0 0},clip]{figures/vae-gon/celeba.png}
        \caption{Samples from a convolutional variational GON trained on CelebA.}
        \label{fig:celeba-vgon}
    \end{subfigure}
    \caption{GONs are able to represent high resolution complex datasets to a high degree of fidelity.}
\end{figure}

\section{Discussion}

Despite similarities with autoencoder approaches, the absence of an encoding network offers several advantages. VAEs with overpowered decoders are known to ignore the latent variables \citep{Chen2017VariationalLossyAutoencoder} whereas GONs only have one network that equally serves both encoding and decoding functionality. Designing inference networks for complicated decoders is not a trivial task \citep{vahdat2020NVAE}, however, inferring latent variables using a GON simplifies this procedure. Similar to GONs, Normalizing Flow methods are also capable of encoding and decoding with a single set of weights, however, they achieve this by restricting the network to be invertible. This requires considerable architectural restrictions that affect performance, make them less parameter efficient, and are unable to reduce the dimension of the data \citep{kingma2018glow}. Similarly, autoencoders with tied weights also encode and decode with a single set of weights by using the transpose of the encoder's weight matrices in the decoder; this, however, is only applicable to simple architectures. GONs on the other hand use gradients as encodings which allow arbitrary functions to be used.

A number of previous works have used gradient-based computations to learn latent vectors however as far as we are aware, we are the first to use a single gradient step jointly optimised with the feedforward pass, making it fundamentally different to these approaches. Latent encodings have been estimated for pre-trained generative models without encoders, namely Generative Adversarial Networks, using approaches such as standard gradient descent \citep{lipton2017precise, zhu2016generative}. A number of approaches have trained generative models directly with gradient-based inference \citep{Han2017Alternating, bojanowski2018optimizing, Zadeh2019VariationalAutoDecoder}; these assign latent vectors to data points and jointly learn them with the network parameters through standard gradient descent or Langevin dynamics. This is very slow, however, and convergence for unseen data samples is not guaranteed. Short run MCMC has also been applied \citep{Nijkamp2020LearningMultilayerLatent} however this still requires approximately 25 update steps. Since GONs train end-to-end, the optimiser can make use of the second order derivatives to allow for inference in a single step. Also of relevance is model-agnostic meta-learning \citep{finn2017model}, which trains an architecture so that a few gradient descent steps are all that are necessary to new tasks. This is achieved by backpropagating through these gradients, similar to GONs.

In the case of implicit GONs, the integration terms in Equation \ref{eqn:implicit-gon-loss} result in computation time that scales in proportion with the data dimension. This makes training slower for very high dimensional data, although we have not yet investigated Monte Carlo integration. In general GONs are stable and consistent, capable of generating quality samples with an exceptionally small number of parameters, and converge to diverse results with few iterations. Nevertheless, there are avenues to explore so as to improve the quality of samples and scale to larger datasets. In particular, it would be beneficial to focus on how to better sample these models, perform formal analysis on the gradients, and investigate whether the distance function could be improved to better capture fine details.

\section*{Conclusion}

In conclusion, we have introduced a method based on empirical Bayes which computes the gradient of the data fitting loss with respect to the origin, and then jointly fits the data while learning this new point of reference in the latent space. The results show that this approach is able to represent datasets using a small number of parameters with a simple overall architecture, which has advantages in applications such as implicit representation networks. GONs are shown to converge faster with lower overall reconstruction loss than autoencoders, using the exact same architecture but without the encoder. Experiments show that the choice of non-linearity is important, as the network derivative jointly acts as the encoding function.

redpajama_set_name: RedPajamaArXiv


im new to this forum, a friend of mine explained the arduino would be to tool of choice for my project but im struggling to understand what i need to type in order to make this work.. clearly not a programmer - im a chef.
so, long story short, i want to make salami.
i have an old fridge, a humidifier, an exhaust fan, a 100w light, a LCD arduino control panel and the ardiuno. i think i need to get a switch board too, but im not too sure.
turn the exhaust on every 4 hours for 30secs to evacuate the chamber of any fermentation gasses.
i dont mind if i use the control panel to change the peramiters daily for the 7 day period because some thicker sausage might take longer and thinner less time. but i do need, lets say if i set it to 18C @ 80% to work its magic and get it to the appropriate temp/humidity. if its 35C outside it will cool. if its 5C outside it will heat using the 100w light..
That sounds like a really fun, kinda big project!
This https://www.sparkfun.com/products/14236 will let you turn on and off your cooler/light/humidifier/fan from the Arduino. (I think this is what you meant by "switch board"?) Unfortunately, I think you'll need a separate one for each appliance since they all need to be able to be switched independently.
After that, you just need to learn programming!
Seriously though, this sounds pretty do-able, but it is going to take a LOT of learning if you haven't done any coding before. Or bribing, if you have any programmer buddies.
That's very do-able for a beginner with a some time on their hands.
As for what kind of Arduino, really anyone of the basic ones will do. So no need to worry about that.
You'll plug (or wire) your humidifier/fridge/exhaust into the relay, but the relay will only provide power when the Arduino sends a "GO" signal to the relay. So using the Arduino you can control when the devices get power and when they turn off.
In order to know when to turn the humidifier/fridge/exhaust on, of course you'll need some sensors. Again, jaquelbot recommended a good one with lots of documentation (free code) on how to get that to work.
Don't let the coding scare you! There's lots and lots of pre-existing code for these kinds of things, so most likely you wont have to write more than a couple of lines of code yourself. Basically you'll just have to add a few "if-then" statements. i.e. "if" the humidity drops below 80, "then" turn send the on signal to the relay controlling the humidifier. Check out https://www.tutorialspoint.com/cprogram ... t_in_c.htm for solid tutorials on what that looks like.
Also don't be afraid to ask for help. Resources like this forum, stackoverflow, and the arduino forum are really helpful.
Good luck and be sure to share your progress if you do follow thru.


The final installment of Mary Magdalene's words to me for the world. We have much to learn from her experience and wisdom!
with all of creation. And when we remember this truth, when we remember this love…..then we can live in the peaceful harmony that has always been God's intention. It is for this peaceful harmony that we were made and it is to this peaceful harmony that we are called.
Jesus lived this truth and died for this truth and in doing so, became a fully Christed being. As such, we are never separate from the truth of Jesus' teachings, neither are we separate from the loving gaze of Christ that heals and transforms and fans the flame of the Spirit within us that seeks to remember and to be reunited with the truth of this love. The invitation is to open our hearts to the truth in Jesus' teachings and to allow ourselves to be transformed in the fire of Divine love. Then we can live as Jesus lived, love as Jesus loved, know the peace that Jesus knew and in doing so, we can help others find their way home. It is to this that Jesus called and empowered me 2000 years ago and it is to this that I call and commission you. Can you be open to remembering the love that you are and can you be a vessel through which others can remember this truth? It is this remembering that is the key to peace within yourself and the key to peace on earth.
How are you called to know Christ and be Christ in the world?
Mary Magdalene was such a blessed woman, whom Jesus' obviously lovedly and respected. If you are interested in reading a historical novel about her, try reading Mary Magdalene; My Story. It incorporates most of her known history and much of the legend about her and weaves it together into a heart-warming yet believable tale, which attempts to keep the true meaning of the spiritual revolution of which she was such an important part.. More information about the book can be found at marymagdalenemystory.com. Thank you.


With the holiday cheer arriving right around the corner, its best to start thinking about that long deserved holiday vacation for you and your loved ones. Surely, topping your list of local and gorgeous destinations is Big Bear Lake where you can experience a true winter mountain experience located only 2 hours from Los Angeles and 2.5 hours from San Diego. The value of booking early is like shopping for gifts early - it can be cost effective, provide better options and you can even avoid the long lines!
Big Bear Lake offers two full service ski resorts - Bear Mountain and Snow Summit.
With Big Bear Lake nearby, the resorts try to make snow before anyone else in Southern California. Real snow vs. man-made snow - don't worry, you can't tell the difference!
There are various types of lodging in the area: hotels, private rentals, resort lodging, B&Bs, cabins, condos and hostels.
Located at 7,000 feet, Big Bear offers more activities than just skiing or snowboarding - there is hiking, zip lining, horseback riding, off-road, mining for treasure, tours, shopping and dining to enjoy during your vacation.
The third week of December to the third week of February is considered peak winter time, so pack warm clothes!
Always carry chains for your vehicle during the winter months. Safety first!
So now you have fun facts for the winter season! Looking up your lodging via BigBear.com is a great way to start determining where you may like to stay this trip. All you have to do is pick your dates, your amenities and you'll get the selections available during your desired check-in. Keep in mind, many facilities may have a nightly minimum requirement you need to meet during holidays or peak weekends, which may vary from 2 to 4 nights at any one time. All the more reason to seek out the best in Big Bear early to plan a vacation you'll never forget!
Planning a vacation can be time consuming. Whether you prefer to stay close to the ski resorts or the Village area, there may be a better spot if you like having everything within close distance. By booking early, your chance of securing a property in your favorite location doubles. Don't forget, the popularity of winter and Big Bear's close proximity to so many California residents does make finding last minute lodging difficult, so plan ahead and book at least 60-90 days before a major holiday in order to avoid paying the higher prices or taking longer minimum stays and finding the right type of property of your choice.
There are still plenty of options available to stay during this winter season and holiday time. It's time you booked online and get started on planning your dream trip to Big Bear Lake!


text:
Wednesday 17 July 2019 | UK News feed
Neil Heywood mystery: Gilded lifestyle of murder suspect's son Bo Guagua
New pictures have cast a fascinating light on the privileged lifestyle and influential circle of friends surrounding the son of ousted Chinese Communist party leader Bo Xilai.
Bo Guagua with Terry Oh (yellow top) and friends
Bo Guagua; with Terry Oh and friend at the Beijing Olympics
Bo Guagua riding in Chongqing
Bo Guagua with Terry Oh at the Great Wall of China
By Josie Ensor, Patrick Sawer, and Richard Eden
9:00PM BST 14 Apr 2012
A prime seat at the Beijing Olympics, nights out with Oxford friends, and a mentor at the heart of the British establishment – this is the gilded lifestyle of the son of the ousted Chinese Communist party leader linked to the death of a British businessman.
These pictures shine new light on the links between Bo Guagua and his Oxford contemporaries, several of whom are now on their way to positions of power and influence within some of the world's leading corporations.
For Bo Guagua, 24, being the scion of one of China's most powerful couples was not only an opportunity to enjoy the finer things in life – it also allowed him to build a network of friends with an eye to future business ventures in his homeland's vast and lucrative markets.
Among his contacts was a key figure in Britain's modern political Establishment - Lord Powell of Bayswater, a former private secretary to both Margaret Thatcher and John Major.
Since leaving the civil service, Lord Powell, whose brother, Jonathan, was Tony Blair's chief of staff, has cultivated strong links with Chinese businessmen and political leaders.
Bo Xilai's son 'escorted from his home by US officials'
Neil Heywood's widow 'silenced by Chinese police'
Neil Heywood 'gave Bo Xilai's financial secrets to lawyer before his death'
Neil Heywood killed 'for threatening to expose money trail'
Neil Heywood: Bo Xilai 'was torn over investigation into wife'
As one Chinese businessman who knows Bo Guagua well said yesterday: "He wants to make a billion dollars and be politically important."
But Bo Guagua and his parents now face a very uncertain future. His mother, Gu Kailai, languishes in detention following her arrest last Tuesday on suspicion of the murder of Neil Heywood, her husband Bo Xilai's close business associate. It was initially claimed that Mr Heywood had died of alcohol poisoning in a hotel room the city of Chongqing last November, but the Chinese government now says Mrs Gu is "highly suspected" of having murdered him following a "conflict over economic interests".
Bo Xilai himself has been removed from his position on the Chinese Communist Party's Politburo and Central Committee, and faces political and personal ruin.
As well as sparking China's biggest political crisis in decades, the couple's downfall has brought attention to the wealth and influence they garnered at home and abroad.
It comes at the end of a week of dramatic twists which have seen new revelations about the extent of Mr Heywood's influence in China, before an apparent rift with Mrs Gu culminated in his death and fears for the safety of his widow Wang Lulu and their two children.
Taken before Mr Heywood's death, these pictures indicate that Bo Guagua spent much of his time at university acquiring a coterie of friends and contacts whose future careers could prove invaluable to his family's business interests.
Though the party's ideology dictates that its leaders live a humble life, with Mr Bo supposedly paid a salary of £300 a month, the wealth they actually acquired through the lucrative business dealings arranged by the likes of Mr Heywood allowed their son access to the best education the West has to offer.
At the age of 12, Bo Guagua was sent by his parents to Papplewick prep school, near Ascot, and then on to Harrow – where the fees are £10,000 a term. The school had been 41-year-old Mr Heywood's alma mater, and he told friends he had helped the boy secure his place there.
In 2006, by which time his father had risen to become China's commerce minister, Bo Guagua went to Oxford University to study philosophy, politics and economics at Balliol College.
He is now a graduate student at Harvard University's John F. Kennedy School of Government, where he has been studying public policy and from where US officials smuggled him into hiding last Thursday night.
At Harvard he became known for his consummate networking skills, helping organise a "China Trek" for Kennedy students last year, where they met senior Politburo dignitaries.
The party visited Chongqing, where Bo Xilai was then head of the Communist Party, and were surprised to be greeted by a police motorcade.
"Everyone knew [Bo Guagua] was somebody important because of the meetings he arranged and also the long police escort," a classmate said.
With his mother now facing a death sentence if found guilty of Mr Heywood's murder and his father stripped of political power, Bo Bo Guagua is now expected to apply for asylum in America.
The future must have looked very different in May 2008, when he posed for photographs with a group of five friends from Oxford.
With him was Lawrence Barclay, 24, an Old Etonian now working as a consultant for the Monitor Group, a multinational management consulting firm with a reputation for secrecy over its clients, which carried out work for the regime of Libyan dictator Muammar Gaddafi before his overthrow last year.
Alongside the pair were fellow Oxford students Augustus Robinson, Pippa Lamb, Terry Oh and Lara Adamson.
Two years after the picture was taken Miss Lamb, 24, who studied Chinese at Oxford, began a six-month stint at the British Embassy in Beijing, working on projects to boost trade links between Britain and China.
She now works for JP Morgan, which has significant investments throughout China.
Mr Robinson, 23, who also studied Chinese, and Lara Adamson, 25, are now trainee solicitors with major law firms.
Terry Oh, a South Korean who was with Bo Bo Guagua at Harrow, is an aspiring male model and also works as an associate for Pimco, the world's largest bond fund, which has extensive interests in south east Asia.
Bo Guagua and Mr Oh attended the 2008 Beijing Olympics together, enjoying the benefit of prestigious seats at the opening ceremony of the games.
They travelled to China again in July last year, visiting the Great Wall and horse riding together in the countryside around Chongqing, the city ruled by Bo Guagua's father until his downfall.
His academic achievements were less than glittering. He was reportedly "rusticated", or suspended, from Balliol for not studying hard enough and had to return later to complete his degree.
He did, however, impress his fellow students when he organised a ball which included a display by Shaolin temple monks and later managed to arrange for Jackie Chan, the Hong Kong-born film star, to address the Oxford Union.
While at Harrow and Oxford university Bo Guagua was mentored by Lord Powell, who is said to have been a "father figure" to the young man. The peer, a president of the China-Britain Business Council who has travelled extensively in the region, declined to elaborate but told The Sunday Telegraph: "I know the boy through his father and saw him from time to time when he was studying at Oxford."
Intriguingly, given rumours that Mr Heywood had links with MI6, Lord Powell is also on the board of Diligence, an intelligence firm set up in 2000 by former members of the British and American secret services, including Nick Day, a former MI5 official. The chairman of the board is William Webster, the former director of the CIA and the FBI.
Mr Heywood is known to have been in the southwestern city of Chongqing last June, when Bo Xilai hosted celebrations as party secretary, and where guests at one event included Lord Powell. A day later, Bo Xilai entertained Lord Mandelson at a banquet in the city.
The arrest of Bo Guagua's mother has also shone a spotlight on her business activities over the past two decades, and in particular her links to Britain. It has now emerged that Mrs Gu stands accused by her husband's former police chief, Wang Lijun, of illegally transferring hundreds of millions of dollars abroad.
It was Mr Wang who had already accused Mrs Gu of poisoning Mr Heywood. In his capacity as a "Bai Shoutao" – literally a "white glove" – Mr Heywood had brokered a series of secretive business deals for Mr Bo, who as a powerful politician was not supposed to get his hands dirty with commercial ventures.
The Sunday Telegraph has now traced some of Mrs Gu's own business activities to an anonymous, run-down building a stone's throw from Bournemouth beach. It was here, in December 2000, she first tried establish a commercial base in the UK.
Mrs Gu had already achieved a reputation as a legal high-flier when, in 1997, she successfully represented a number of Chinese companies in a $1 million bankruptcy dispute with an American firm.
Three years later came her move into Britain, where she teamed up with a French architect called Patrick Devillers. Mr Devillers, who is understood to be married to a Chinese woman, agreed to set up a company with Mrs Gu.
The move coincided with the enrolment of her son at Papplewick school, which charges fees of around £22,400 year. There is now speculation that Mrs Gu may have set up firms in Britain in order to channel the money Mr Heywood was making for her and her husband in China.
Indeed, an article last Wednesday in the People's Daily, the Communist Party's official newspaper – though ostensibly about the wider issue of corruption – was widely interpreted as a further attack on Mr Bo and others like him. It pointedly stated: "Some people had secretly gained dual citizenship and foreign identities, transferred money and goods overseas, used relatives, friends and mistresses to conceal their wealth."
Mrs Gu and Mr Devillers established their firm, called Adad Ltd, at a business park in Poole, with a working capital of just £1,000. It described itself in the records of Companies House as offering "service activities" for other companies, with Mr Devillers and Mrs Gu – using her business name of Horus Kai – as its sole directors.
Adad appears to have operated as no more than a "shell" company, dissolving in 2003 having filed no accounts during its three years of operation, and could have provided a front for the movement of funds from China.
Mrs Gu and Mr Devillers gave their residential address as the top floor terrace apartment of Keystone House, six miles away in Bournemouth. Office workers in the building were surprised that a woman with Mrs Gu's wealth and connections could have lived in the flat, as it is usually rented by students from Bournemouth University looking for cheap accommodation.
Following the dissolution of Adad, the trail of Mrs Gu's affairs becomes even more complex.
By 2006 Mr Devillers, who also occupied a smart apartment in a residential block in Earl's Court, west London, had moved to Beijing, where it is thought he continued to pursue their joint business interests.
He registered the Chinese arm of their operations through the Beijing Angdao Law Firm, based at a smart address in an apartment complex overlooking the Chinese capital's Olympic Park and iconic Bird's Nest Stadium.
Mr Devillers is thought to have moved back to France in 2008. Angdao Law's boss, who gave his name only as Mr Li and described himself as friend of Mr Devillers, said: "He no longer works here, or visits. I think he went back to France."
In the meantime, Mrs Gu continued to pursue her work through Angdao Law, introducing foreign business clients to China's senior party leaders and further enriching herself and her husband.
Mr Heywood met Mr Bo in the 1990s after writing to him to ask for help pursuing business opportunities in China and promising to bring foreign investment to Chongqing.
He soon became part of Bo Xilai's trusted inner circle and as such managed to broker a deal between the Chinese authorities and the British government for the release of Matthew Bennett, a senior executive with Aston Martin, the luxury car manufacturer, who had been prevented from leaving the country for almost seven months in 2007 following a trade dispute.
The Old Harrovian is understood to have grown concerned about his safety after a dispute with Mrs Gu last year, in which she suspected that he was preparing to reveal details of the family's business activities. Shortly before his death, Mr Heywood is thought to have handed the dossier to his British lawyer as his "insurance policy", in case the worst should happen to him. Its actual contents – along with the identity of the lawyer – remains a mystery. But one thing is in little doubt - such an act would have been regarded as a gross betrayal.
* Additional reporting by Peter Simpson in Beijing, Malcolm Moore in Chongqing and Peter Allen in Besancon
World News »
In UK News
rugbyTelegraph View
Telegraph Obituaries
Latest Politics News
Up Helly Aa Festival
Aviemore Sled Dog Rally
Patrick Sawer,"; (function(){ function loadChartbeat() { window._sf_endpt=(new Date()).getTime(); var e = document.createElement('script'); e.setAttribute('language', 'javascript'); e.setAttribute('type', 'text/javascript'); e.setAttribute('src', (("https:" == document.location.protocol) ? "https://a248.e.akamai.net/chartbeat.download.akamai.com/102508/" : "http://") + "static.chartbeat.com/js/chartbeat.js"); document.body.appendChild(e); } var oldonload = window.onload; window.onload = (typeof window.onload != 'function') ? loadChartbeat : function() { oldonload(); loadChartbeat(); }; })();

redpajama_set_name: RedPajamaCommonCrawl


It incorporates a high-quality cardioid condenser microphone capsule for great quality audio recordings when used with a wide range of cameras. A directional microphone, the VideoMicro reduces distracting peripheral sounds and focusses on the audio in front of the camera. Its pickup is more forgiving than RØDE's other on-camera microphones providing a more natural sound when recording indoors. Perfect for capturing incredible audio to accompany inspiring vision. The microphone body is made from aluminum, giving it a high level of RF rejection, and is finished in RØDE's high-grade ceramic anti-glare black coating. Included with the VideoMicro is a camera shoe mount featuring a Rycote Lyre shock mount.


text:
Supplementary Components1
Supplementary Components1. essential for life; however, it also presents a leading source of mutation and genomic instability that can cause systemic diseases such as malignancy (Tomasetti et al., 2017; Tubbs and Nussenzweig, 2017). The progression of tens of thousands of replication forks in the cell can MD2-TLR4-IN-1 be challenged by many impediments such as insufficient nucleotides, DNA lesions, secondary structures (e.g. G-quadruplexes and hairpins) and collisions with the transcription apparatus (Zeman and Cimprich, 2014). Oncogene activation also induces replication stress that threatens genome stability and fuels tumorigenesis (Macheret and Halazonetis, 2015). The presence of these challenges necessitates mechanisms that preserve the integrity of the fork structure under stress in order to complete replication with high fidelity in each cell cycle. Due to the presence of single-stranded DNA and DNA ends in the structure, replication forks are intrinsically vulnerable to nucleolytic attack, especially COG3 in the event of replication stress (Berti and Vindigni, 2016; Branzei and Foiani, 2010). A key pathway for fork protection is the ATR-Chk1-dependent replication checkpoint. Beyond its canonical function in halting the cell cycle to allow time for repair, the checkpoint pathway also directly protects fork structure and promotes fork restart in response to replication stress (Saldivar et al., 2017; Yazinski and Zou, 2016). Studies in yeast and mammalian cells indicate that a crucial function of the replication checkpoint is usually to restrain or eliminate the activity of Exo1, a 5-to-3 exonuclease that can process fork structure through resection of DNA ends (Cotta-Ramusino et MD2-TLR4-IN-1 al., 2005; El-Shemerly et al., 2008; Segurado and Diffley, 2008). Although a proper function of Exo1 is certainly very important to multiple pathways of DNA fix including mismatch fix and DNA double-strand break (DSB) fix, uncontrolled Exo1 activity during replication could cause extreme fork resection, chromosomal instability and decreased cell viability upon replication tension (Cotta-Ramusino et al., 2005; Engels et al., 2011; Keijzers et al., 2016; Segurado and Diffley, 2008). In fungus, treatment with hydroxyurea (HU) network marketing leads to Rad53 (useful ortholog of Chk1)-reliant phosphorylation of Exo1, leading to attenuation of its activity in resection (Morin et al., 2008). In individual cells, Exo1 is certainly phosphorylated within an ATR-dependent way after extended replication tension, resulting in Exo1 degradation and ubiquitination, thereby staying away from aberrant fork resection (El-Shemerly et al., 2008). Furthermore to checkpoint elements, the adaptor proteins 14-3-3s have already been proven to prevent aberrant fork resection by Exo1, although the complete mechanism is MD2-TLR4-IN-1 certainly yet to become described (Engels et al., 2011). Several various other elements, such as BRCA1, BRCA2, BARD1, PALB2, Rad51, MD2-TLR4-IN-1 Rad51 paralogs, FANCA, FANCD2, FANCJ, BOD1L, WRNIP1, RECQ1, PARP1, Abro1, CtIP, AND-1 and SETD1A, have also been shown to prevent fork degradation, likely by surpressing the function of Mre11, Dna2 or Exo1 nucleases directly at the fork (Abe et al., 2018; Billing et al., 2018; Cotta-Ramusino et al., 2005; Engels et al., 2011; Hashimoto et al., 2010; Higgs et al., 2015; Higgs et al., 2018; Iannascoli et al., 2015; Karanja et al., 2014; MD2-TLR4-IN-1 Keijzers et al., 2016; Lemacon et al., 2017; Leuzzi et al., 2016; Lomonosov et al., 2003; Mijic et al., 2017; Peng et al., 2018; Petermann et al., 2010; Przetocka et al.,.
CategoriesNicotinic Acid Receptors
Previous PostPrevious Over-activation of autophagy due to increased degrees of reactive air species is an integral system of lipopolysaccharide-induced acute lung damage
Next PostNext Gastrointestinal cancers remain a tremendous burden in society