text:
Herve Caen takes the gloves off View next topic
Duck and Cover Forum Index » In The News
King of Creation
Righteous Subjugator
[ Company -> Article ] - News related to Top Story: Interplay's legal argument summarized | More info on Person: Herve Caen | More info on Company: Bethesda v. Interplay
Duck and Cover has obtained Interplay's latest court filing in its battle with Bethesda. This filing is a 17 page statement from Interplay CEO Herve Caen, and he is not very happy at all with Bethesda's conduct during the case. The filing mostly summarizes Interplay's legal arguments, but also allows Herve Caen to publically take shots at Bethesda. We also learn a bit more about Fallout Online's development status, including how fleshed out the game world is. Read on!
First, some background:
Interplay has continuously sold Fallout, Fallout 2, and Fallout Tactics since their initial release dates. Attached hereto as Exhibits "1"-"3" are true and correct copies of the box cover art used by Interplay with respect to such games for many years. At various times, Interplay also has sold compilations of two or all three of the titles. For example, Interplay sold Saga Fallout, a compilation of Fallout, Fallout 2, and Fallout Tactics, beginning in or about 2004. The same three-game-combo was distributed as Fallout Trilogy, Fallout Collection, or Fallout: The Ultimate Collection at least as early as about 2005 or 2006. Interplay has also sold a compilation called Fallout: Radioactive Collection.
On April 4, 2007, Interplay and Bethesda entered into an Asset Purchase Agreement (the "APA") releated to the Fallout intellectual property, which superseded the Exclusive License [the exclusive license agreement licesed the rights to make Fallout 3, 4, and 5 to Bethesda]. Under the APA, Interplay transferred the Fallout mark and related intellectual property to Bethesda, in consideration of Interplay receiving a certain cash payment and the perpetual, royalty-free, worldwide, and exclusive right to continue selling the Interplay-developed Fallout games, which consisted of Fallout, Fallout 2, Fallout Tactics and Fallout: Brotherhood of Steel, referred to collectively in the APA as the "Pre-existing Fallout Games."
Interplay retained the rights to "exclusively manufacture, have manufactured, sell and distribute" the Pre-existing Fallout Games (the "Merchandising Rights")....Interplay would never have entered into the APA unless it received the perpetual Merchandising Rights, as part of the considertaion for entering into said Agreement...If you takeaway the rights granted to Interplay, then a material portion of the consideration that Interplay was to receive in exchange for the transfer of rights to Bethesda is substantially, if not totally, eliminated.
In conjunction with the APA, Interplay and Bethesda also executed a Trademark License Agreement ("TLA"). (Exhibit "D" to Bethesda's Motion) The TLA conditionally licensed back to Interplay the right to create the Fallout massively multiplayer online game ("MMOG") that it had reserved for itself under the superceded Exclusive License. In exchange for the license, Interplay agreed to pay a royalty to Bethesda based on game sales, all as more particularly provided in the TLA. To retain the Fallout MMOG rights, Interplay had to begin development and secure development financing within two years of the effective date of the TLA. Nothing in the APA, however, requires Interplay to spend any particular amount of money to develop the MMOG. Rather Interplay must deliver a finished product approved by Bethesda within a certain period of time.
One of the major issues in the whole court case is the language used in the contracts. The agreements specify that Interplay had to begin full-scale development of the Fallout MMO by a certain date, as well as secure a certain amount of funding by a certain date. The language of these statements in the contracts is misleading, Caen claims. The two parties have two very different interpretations. Here is Herve Caen's:
The TLA requires Interplay to begin "full-scale development within two years of its effective date, April 4, 2007, but does not define what this means.
To create its MMOG, Interplay spent more than a year in pre-production. It hired the game designers from the original Fallout game to work on the game design. Interplay hired artists to create concept art for reference during production...Interplay also hired writers to work on the MMOG's story line. The current game design document is more than 600 pages long and describes background material, such as the game world, characters, creatures, weapons, equipment, back story and other key concepts concerning game play.
In conjunction with Earthrise, Masthead has already developed a proprietary MMOG engine and software tool sets over a period of several years and at an equivalent cost of several million U.S. dollars. Masthead has also developed its own networking software to permit thousands of simultaneous connections to and interactions with the game world. Interplay licensed Masthead's technology and hired it as a contractor to perform software programming and some game design services under Interplay's direction and control.
Bethesda also takes issue with the fact that Interplay is working with Masthead in creating Fallout Online. Bethesda apparently claims that this is tantamount to sublicensing the rights to a third party, something Interplay is not allowed to do. Herve Caen completely disagrees:
Interplay has not sub-licensed its rights to Masthead, but rather entered into a development agreement for Masthead to work with Interplay to develop the MMOG. Interplay has not assigned any rights in the Fallout mark or in its MMOG to Masthead or to anyone.
We also have confirmation that there is a playable build of the game:
The advanced state of Masthead's technology and Interplay's extensive pre-production phase resulted in a playable demonstration in March 2009. The demonstration is a portion of the post-apocalyptic game world, incorporating buildings, locations, objects, non-player characters, and controllable player characters. At the time of the hearing, or if requested by the Court prior thereto, we will submit the video demonstration of the game from March 2009.
As far as Interplay's perspective on full-scale development:
Interplay and Masthead currently have a number of artists, programmers, writers, and game designers working on this project so that Interplay can timely meet its deliverables as required by the TLA. I believe that Interplay has commenced what I would term "full-scale" development of its MMOG.
Another issue Interplay has with Bethesda's conduct is the fact that Bethesda has consistently denied them the use of the Fallout name in promoting their Fallout MMO:
Interplay has never used the Fallout mark in connection with the development of its MMOG since the Parties executed the APA and TLA, and has never used the Fallout mark in commerce related to its MMOG, which remains in development and has not been commercially released.
In fact, on April 3, 2008, Interplay sought approval to use the Fallout mark on its website to announce the MMOG project that it was working on in connection with the TLA...In a nutshull, Bethesda refused to permit Interplay to announce its Fallout MMOG even though it was entitled to make the announcement pursuant to the TLA. Interplay should have been, and should be, allowed to use the Fallout mark while it develops its MMOG, and the refusal of Bethesda to permit this is in bad faith.
Interplay, however, apparently showed some good faith:
The TLA does not specify any mechanism by which Interplay must communicate to Bethesda that it has satified the conditions of the TLA or that Interplay has to do this at all. Notwithstanding, Interplay voluntarily sent a letter to Bethesda on April 2, 2009 informing Bethesda that Interplay had satisfied the conditions of the TLA.
And here is where the gloves really come off:
It is interesting that in or about August 2007, just four months after Interplay had entered into the TLA with Bethesda, Bethesda's parent compabny, ZeniMax Media LLC, announced the creation of its own MMOG development house, ZeniMax Online Studios. Based on Bethesda's subsequent conduct it thus appears that Bethesda never intended to permit Interplay to develop an MMOG under the terms of the TLA and planned from the outset to terminate Interplay's rights and grant them instead to affiliated ZeniMax Online Studios for a Fallout MMOG.
Bethesda's attempt to terminate the MMOG rights undert the TLA is consistent with its overall conduct designed to interfere with Interplay's exploitation of rights granted undert the parties' agreements, including Interplay's Merchandising Rights to the Interplay-developed Pre-existing Fallout Games.
We have since learned that ZeniMax Online is most likely working on an Elder Scrolls Online game, not a Fallout MMO. Bethesda stated in court that they are NOT working on a Fallout MMO at this time.
One of the issues Bethesda had going into the preliminary injunction hearing was the fact that Interplay has been distributing the original Fallout games on services such as Steam and GOG. Interplay believes that they are well within their rights to distribute these games on these services, despite Bethesda's protestions, based on the following:
Nothing in the APA requires Interplay to seek permission to distribute the Pre-existing Fallout Games through third party distributors. In fact, the APA at Section 5.10 specifically states that Interplay has the right to "have manufactured" the Interplay-developed games, acknowledging that sublicenses to third parties in the context of distribution are intended.
The rest of the 17 page filing is mostly a rehash of what we already know and not worth posting again. From all of the filings that we have seen, Bethesda is being made out as the clear-cut bad guy trying to cheat the little guy. While this might not be the case, we haven't seen much from Bethesda's point of view on the matter and are, therefore, left to consider them as the aggressor. In a land of corporate legal departments and PR machines, we often find ourselves rooting for the little guy without really knowing both sides. While the Interplay v. Bethesda case is not really a case of David v. Goliath, we can't really help but to draw comparisons.
Anyways, it's all the Fallout community has to read about right now!
SenisterDenister
250 Posts til Somewhere
Location: Cackalackyland
Bethesda's fallout can't kill the cockroach.
Ausir
The Vault Overseer
Here's a summary with some more details, as well as quotes from correspondence between Herve and Bethesda's reps:
http://fallout.wikia.com/wiki/User_blog:Ausir/Herve_Caen%27s_court_statement_revealed
http://fallout.wikia.com/wiki/User_blog:Ausir/Interplay-Bethesda_correspondence_unveiled
lyssagale12
Location: losangeles
Ausir wrote:
That's cool........! nice link u have shared........ I appreciate with that.
Thanks for useful info
Wastelander's Nightmare
Location: Drifting Onward
Rapid ban, go!
jetbaby
Mamma's Gang member
Location: Magical Island
But I need more handbags, damn it!

redpajama_set_name: RedPajamaCommonCrawl


text:
\subsubsection*{Acknowledgements}
\bibliographystyle{apalike}

\section{Algorithm Details}
\label{sec:algorithm}
\glsresetall

Here we provide the details on the \gls{ACI} algorithm.
We start by specifying the class of GP models that we consider in our framework. We then discuss the technical details of \gls{ACI} and proof that the automatically constructed augmentation indeed leads to a conditionally conjugate model.

\parhead{GP Models with a super-Gaussian likelihood.}
  \gls{ACI} can be applied to GP models, where the likelihood is within the class of super-Gaussian likelihoods. A super-Gaussian likelihood is of the form
	\begin{align}
	p(\boldsymbol{y}|\boldsymbol{f};\theta) =& C(\theta)e^{g(\boldsymbol{y};\theta)^\top \boldsymbol{f}}\varphi(||h(\boldsymbol{f},\boldsymbol{y})||_2^2), \label{eq:likelihood}
	\end{align}
	where $\theta$ are hyperparameters of the likelihood, $C(\theta)$ is the normalizing constant, $g(\boldsymbol{y};\theta)$ is an arbitrary function, $\varphi$ is a \emph{\gls{PDR} function}\footnote{$\varphi$ is a positive definite radial function if $\varphi(r)$ is completely monotone for all $r\ge 0$ and $\lim_{r\rightarrow 0} \varphi(r)= 1$.},

	and $h$ is a linear function in $\boldsymbol{f}$, such that we can write
	\begin{align}
	||h(\boldsymbol{f},\boldsymbol{y})||_2^2 =& \alpha(\boldsymbol{y},\theta) - \beta(\boldsymbol{y},\theta)^\top \boldsymbol{f} + \gamma(\boldsymbol{y},\theta)||\boldsymbol{f}||^2_2, \label{eq:h}
	\end{align}
	where $\alpha,\beta,\gamma$ are arbitrary functions.
	We omit $\theta$ in the later derivations for clarity.

  Many interesting models

  are instances of super-Gaussian likelihood GP models.
  In Table~\ref{tab:likelihoods}, we present several likelihood functions with their corresponding parameter settings of the super-Gaussian likelihood as given in Eq.~\ref{eq:likelihood}.

  \input{input_likelihood_table}

  \parhead{Constructing new likelihoods.} Using Eq.~\ref{eq:likelihood}, we can also construct novel likelihood functions based on existing kernel functions. In this paper we propose the Matern 3/2 likelihood.

 %
 %

\subsection{Step 1: Conjugate augmentation}
\label{subsec:augmentation}
Given the likelihood of the model, \gls{ACI} constructs a conditionally conjugate auxiliary variable augmentation as follows.
We first define a family of distribution $\pi_\varphi(\omega|c)$, which will be useful for constructing the augmentation.

For the case $c=0$, the distribution $\pi_\varphi(\omega|0)$ is defined by the inverse Laplace transform of $\varphi(\cdot)$,
\begin{align}
\pi_\varphi(\omega|0) = \mathcal{L}^{-1}\left\{\varphi(\cdot)\right\}(\omega).\label{eq:invlapomega}
\end{align}

The inverse Laplace is the inverse mapping of the Laplace transformation and can be computed by the Bromwich integral formula\footnote{The inverse Laplace transformation of a function $\varphi(\cdot)$ can be computed by $\mathcal{L}^{-1}\left\{\varphi(\cdot)\right\}(\omega) = \lim_{T\rightarrow\infty} \frac{1}{2\pi i}\int_{b-iT}^{b+iT} e^{r\omega}\varphi(r)dr$, where $b$ can be arbitrarily chosen but has to be larger than the real part of all singularities of $\varphi$.} \citep{debnath2014integral} and it defines a valid density in our setting (see proof of Theorem~\ref{thm:augmentation}).
Remarkably, we will see that for the final updates of our algorithm, we do not need to compute the inverse Laplace transformation explicitly.

We generalize the base distribution $\pi_\varphi(\omega|0)$ by applying an exponential tilting:
\begin{align}
\pi_\varphi(\omega|c) = \frac{e^{-c^2 \omega}\pi_\varphi(\omega|0)}{\varphi(c^2)}, \label{eq:piomega}
\end{align}
where $c\in \mathbb{R}$.

\begin{theorem} A GP model with a super-Gaussian likelihood (of the form of Eq.~\ref{eq:likelihood}) is rendered \textbf{conditionally conjugate} by the auxiliary variable augmentation ${p(\boldsymbol{y},\boldsymbol{f},\boldsymbol{\omega};\theta) = p(\boldsymbol{y}|\boldsymbol{f},\boldsymbol{\omega};\theta) p(\boldsymbol{f}) p(\boldsymbol{\omega})}$. The augmented likelihood is
\begin{align*}
	p(\boldsymbol{y}|\boldsymbol{f},\boldsymbol{\omega};\theta) &= C(\theta)\exp\left(g(\boldsymbol{y};\theta)^\top \boldsymbol{f} - ||h(\boldsymbol{f},\boldsymbol{y})||_2^2\boldsymbol{\omega}\right)
\end{align*}
and the prior distribution of the auxiliary variables is
\begin{align*}
  p(\boldsymbol{\omega}) =\pi_{\varphi}\left(\boldsymbol{\omega}| 0\right).
\end{align*}
\label{thm:augmentation}
\end{theorem}
\parhead{Proof:}
We first apply Schoenberg's theorem \citep{schoenberg1938metric}, which states that a function $\mathbb{R}^d\ni \boldsymbol{x}\rightarrow \varphi(\|\boldsymbol{x}\|_2^2)$ is a \gls{PDR} function for any dimension $d>0$ if and only if $\varphi(r)$ is a completely monotone function on the domain $r\geq0$.

A completely monotone function $\varphi(\cdot)$ has the property that it is infinitely differentiable and its derivatives have an alternating sign \citep{bernstein1929fonctions}, i.e.
\begin{align}
	(-1)^k\varphi^{(k)}(r) > 0,\quad r\in[0,+\infty),\; k=0,1,2,\ldots.
\end{align}
As a direct consequence, $\varphi(\cdot)$ is a positive, decreasing, and convex function and the first derivative of $\varphi(\cdot)$ is a concave function.

Building on these properties, \citet{widder1946laplace}~states that we can rewrite $\varphi(\|h(f,y)\|_2^2)$ as a Gaussian scale-mixture
\begin{align}
\varphi\left(\|h(f,y)\|_2^2\right)= \int_0^\infty e^{-\|h(f,y)\|_2^2 \omega} d\mu(\omega),\label{eq:scale-mixture}
\end{align}
with respect to a Borel measure $\mu(\omega)$.
We apply the monotone convergence theorem \citep{yeh2006real}, which gives that $\mu(\omega)$ is even a probability measure iff $\lim_{r\rightarrow 0}\varphi(r)=1$. Since we have a probability measure, we write $d\mu(\omega) = p(\omega)d\omega$ and which leads to the equality $\varphi(r)=\laplace{p(\omega)}{r}$, where $\mathcal{L}$ denotes the Laplace transformation. The inverse Laplace transformation gives the density of the auxiliary variable ${p(\omega)=\mathcal{L}^{-1}\left\{\varphi(r)\right\}(\omega)=\pi_\varphi(\omega|0)}$.

Therefore we can rewrite the super-Gaussian likelihood Eq. \ref{eq:likelihood} as :
\begin{align}
p(y|f)=C(\theta)\int_0^\infty e^{-g(y)f-\|h(f,y)\|_2^2\omega}p(\omega)d\omega.
\end{align}
Adding the auxiliary variable $\omega$ with prior $p(\omega)$ to the model, we obtain the augmented likelihood	${p(\boldsymbol{y}|\boldsymbol{f},\boldsymbol{\omega};\theta) = C(\theta)\exp\left(g(\boldsymbol{y};\theta)^\top \boldsymbol{f} - ||h(\boldsymbol{f},\boldsymbol{y})||_2^2\boldsymbol{\omega}\right)}$.

Since the function $g(\boldsymbol{y};\theta)^\top \boldsymbol{f} - ||h(\boldsymbol{f},\boldsymbol{y})||_2^2\boldsymbol{\omega}$ is by definition quadratic in $\boldsymbol{f}$ the augmented likelihood is proportional to an (unnormalized) Gaussian distribution in $\boldsymbol{f}$, hence, conditionally conjugate in $\boldsymbol{f}$.\\
For the augmented variable $\omega_i$, the likelihood $p(y|\omega,f)$ act as an exponential tilting of $p(\omega)$ and the full conditional in $\omega$ will stay in the same family of distributions.
QED.

\subsection{Step 2: Complete Conditionals}
\label{subsec:complete_conditionals}
 Since the augmented model (Section~\ref{subsec:augmentation}) is conditionally conjugate, the complete conditional distribution are in the same family as their associated prior distributions and are given in closed form.

\begin{theorem}
  The \textbf{complete conditional distributions} of the augmented model presented in Section~\ref{subsec:augmentation} are given by
  \begin{align}
  \begin{aligned}
    p(\omega_i|f_i,y_i) =& \pi_{\varphi}\left(\omega_i| \|h(f_i,y_i)\|_2\right),\\
    p(\boldsymbol{f}|\boldsymbol{y},\boldsymbol{\omega}) =& \mathcal{N}\left(\boldsymbol{f}| \boldsymbol{\mu} ,\boldsymbol{\Sigma}\right),
  \end{aligned}\label{eq:fullconditionals}
  \end{align}
  where $\boldsymbol{\Sigma} = \left(\mathrm{diag}\left(2\boldsymbol{\omega}\circ \gamma(\boldsymbol{y})\right) + K^{-1}\right)^{-1}$ and $\boldsymbol{\mu} = \boldsymbol{\Sigma}\left(g(\boldsymbol{y})+\boldsymbol{\omega}\circ \beta(\boldsymbol{y}) + K^{-1}\boldsymbol{\mu}_0\right)$, $\circ$ denotes the Hadamard product and the function $h(\cdot)$ is given by the form of likelihood (see  Eq.\ref{eq:h}).
  \label{thm:fullcond}

\end{theorem}\vspace{-0.3cm}
The proof is given in Appendix~\ref{appendix:fullcond}

\subsection{Step 3: Efficient inference}
\label{subsec:inference}
In the final step of our inference pipeline, we leverage the conditionally conjugate structure of the augmented model and derive two inference methods. First, we propose a scalable stochastic variational inference (SVI) method that builds on efficient block coordinate ascent updates (CAVI) updates,  computed in closed form. Second, we develop a Gibbs sampling scheme that generates samples from the exact posterior.

\subsubsection{Augmented variational inference}
\label{subsec:VI}
We implement the classic stochastic variational inference (SVI) algorithm for conditionally conjugate models described by \citet{hoffman2013stochastic}, which builds on block coordinate ascent updates. The updates can be interpreted as natural gradient updates and are much more efficient than ordinary Euclidean gradient updates \citep{amari1998natural}.

\parhead{Variational approximation.}
We approximate the posterior distribution of the latent GP values by
assuming a decoupling between $\boldsymbol{f}$ and $\boldsymbol{\omega}$.
The family of the optimal variational distribution can be easily determined
by averaging the complete conditionals in log-space, as given in Eq.~\ref{eq:cavi_updates}~\citep[see e.g.][]{blei2017variational}.
From the above decoupling assumption, it follows that the optimal variational posterior is in the variational family
\begin{align}
q(\boldsymbol{f},\boldsymbol{\omega}) = q(\boldsymbol{f})\prod_{i=1}^Nq(\omega_i), \label{eq:meanfield}
\end{align}
where $q(\boldsymbol{f})=\mathcal{N}\left(\boldsymbol{f}| \boldsymbol{m},\boldsymbol{S
}\right)$ and $q(\omega_i)=\pi_{\varphi}(\omega_i| c_i)$ and $\boldsymbol{m},\boldsymbol{S}$ and $\boldsymbol{c}$ are the \emph{variational parameters}.

\parhead{Variational updates.}
We start with deriving the variational updates for the variational Gaussian distribution,
\begin{align*}
	q(\boldsymbol{f}) &\propto \exp\left[\expec{q(\boldsymbol{\omega})}{\log p(\boldsymbol{f}|\boldsymbol{\omega},\boldsymbol{y})}\right]\\
	&\propto\exp\left[\sum_i g(y_i) f_i - \|h(f_i,y_i)\|_2^2\expec{q(\omega_i)}{\omega_i}\right]p(\boldsymbol{f})
\end{align*}
Computing the variational updates of $q(\boldsymbol{f})$ boils down to computing the first moment of $\boldsymbol{\omega}$.
Remarkably, the moments
of $\pi_\varphi$ can be computed without computing the closed-form density of $\pi_\varphi$ explicitly,  i.e. without evaluating the inverse Laplace transformation of $\varphi$ (Eq.~\ref{eq:invlapomega}).

The moments can be computed by differentiating the moment generating function, which is itself a Laplace transform.
For our algorithm, we only need the first moment of $\omega$, which is given by
\begin{align*}
\expec{q(\omega)}{\omega} =& \left.\frac{d\laplace{q(\omega)}{-t}}{dt}\right|_{t=0}= -\frac{\varphi'(c^2)}{\varphi(c^2)}=\overline{\omega},
\end{align*}
which can be cheaply computed via automatic differentiation.

The updates for the variational distribution of the auxiliary variables $q(\boldsymbol{\omega})$ are computed as follows.
\begin{align*}
	q(\omega_i)
	\propto \exp\left[ -\expec{q(f_i)}{\|h(f_i,y_i)\|_2^2}\omega_i+\log p(\omega_i)\right]\\
	&\propto \exp\left(-\expec{q(f_i)}{\|h(f_i,y_i)\|_2^2} \omega_i\right)p(\omega_i)\\
	&= \pi_\varphi(\omega_i|\sqrt{\expec{q(f_i)}{h(f_i,y_i)^2}}).\label{eq:vardist}
\end{align*}
We get then the update  ${c_i=\sqrt{\expec{q(f_i)}{\|h(f_i,y_i)\|_2^2}}}$, which can be easily computed in closed form since $\|h(f_i,y_i)\|_2^2$ is a quadratic function of $f_i$.

The coordinate ascent variational inference (CAVI) method is summarized in Algorithm~\ref{alg:VI}.

\input{algo_vi}

\parhead{Sparse GP approximation.}
To scale our method to big datasets, we approximate the latent GP $\boldsymbol{f}$ by a {\it sparse Gaussian process} building on {\it inducing points}.
We introduce $M$ inducing points $\boldsymbol{u}$ and connect the GP values with the inducing points via the joint prior distribution $p(\boldsymbol{f}, \boldsymbol{u})$ given in \citet{titsias2009variational}.
The introduction of inducing points preserves conditional conjugacy and allows for mini-batch sampling of the data (stochastic variational inference). This scales the algorithm to big datasets and has the computational complexity $\mathcal{O}(M^3)$.
The SVI version of our algorithm only slightly changes the updates that are presented in Algorithm~\ref{alg:VI}. It is deferred to Appendix~\ref{appendix:sparse}.

\vspace{-0.3cm}

\subsubsection{Gibbs sampling}
\label{subsec:Gibbs}

To sample from the exact posterior distribution, a Gibbs sampling scheme alternates between sampling from the complete conditional distributions. In the following we propose a sampling scheme for the distribution family $\pi_\varphi(\omega|c)$ that is automatically constructed given the \gls{PDR} function of the likelihood $\varphi(\cdot)$

The distribution class $\pi_\varphi$ is defined in Eq.~ \ref{eq:invlapomega} and is based on the inverse Laplace transform of $\varphi(\cdot)$.
However there is no general approach to compute the inverse Laplace in closed form \citep{cohen2007numerical}.
We circumvent this issue by proposing an algorithm that only evaluates the inverse Laplace transformation point-wise but does not need access to its full analytical form.
We apply the method proposed by \citet{ridout2009generating}, which build on the fact that the \gls{CDF} $F_{\pi_\varphi(\omega|c)}(\cdot)$ can be computed via the inverse Laplace transform of a scaled (forward) Laplace transform,
\begin{align*}
F_{\pi_\varphi(\omega|c)}(x) &= \mathcal{L}^{-1}\left\{\frac{\mathcal{L}\left\{\pi_\varphi(\omega|c)\right\}(s)}{s}\right\}(x)\\
&= \mathcal{L}^{-1}\left\{\frac{\varphi(s+c^2)}{s\varphi(c^2)}\right\}(x).
\end{align*}
To generate samples from $\pi_\varphi(\omega|c)$, we first generate a uniform sample $u\sim\mathcal{U}\left[0,1\right]$ and then push it through the inverse \gls{CDF}, $\omega=F^{-1}_{\pi_\varphi(\omega|c)}(u)$ \citep{devroye1986nonuniform}
Finally, to compute the inverse \gls{CDF}, we solve a fixed point problem using the modified Newton-Raphson method described by \citet{ridout2009generating}. We solve the equation $F_{\varphi(c)}(\omega)=u$ by repeatedly setting $\omega \leftarrow \omega - F_{\varphi(c)}(\omega)/\pi_\varphi(\omega|c)$ until reaching convergence. We numerically approximate the (forward) \gls{CDF} $F_{\varphi(c)}(\omega)$ by the cheap trapezoidal method introduced in \citet{abate2000introduction}, which has error guarantees. The cost of this process is negligible against the matrix inversion for sampling $\boldsymbol{f}$.
\label{sec:gibbs}
All steps are summarized in Algorithm~\ref{alg:Gibbs}.

Note that for some likelihood functions (e.g. the logistic likelihood function), the inverse Laplace transform can be derived analytically and the steps described above can be optimized by using an existing the sampler for the corresponding complete conditional distribution.

\input{algo_gibbs}

\section{Appendix}

\subsection{Proof of theorem~\ref{thm:fullcond}}
\label{appendix:fullcond}
Theorem~\ref{thm:fullcond} states:
\begin{theorem*}
	The \textbf{complete conditional distributions} of the augmented model presented in Section~\ref{subsec:augmentation} are given by
	\begin{align*}
	\begin{aligned}
	p(\omega_i|f_i,y_i) =& \pi_{\varphi}\left(\omega_i| \|h(f_i,y_i)\|_2\right),\\
	p(\boldsymbol{f}|\boldsymbol{y},\boldsymbol{\omega}) =& \mathcal{N}\left(\boldsymbol{f}| \boldsymbol{\mu} ,\boldsymbol{\Sigma}\right),
	\end{aligned}
	\end{align*}
	where $\boldsymbol{\Sigma} = \left(\mathrm{diag}\left(2\boldsymbol{\omega}\circ \gamma(\boldsymbol{y})\right) + K^{-1}\right)^{-1}$ and $\boldsymbol{\mu} = \boldsymbol{\Sigma}\left(g(\boldsymbol{y})+\boldsymbol{\omega}\circ \beta(\boldsymbol{y}) + K^{-1}\boldsymbol{\mu}_0\right)$, $\circ$ denotes the Hadamard product and the function $h(\cdot)$ is given by the form of likelihood (see  Eq.\ref{eq:h}).
\end{theorem*}
\parhead{Proof:}
For the full conditional on $\boldsymbol{f}$:
\begin{align*}
p(\boldsymbol{f}|\boldsymbol{y},\boldsymbol{\omega}) \propto& p(\boldsymbol{y}|\boldsymbol{f},\boldsymbol{\omega})p(\boldsymbol{f})\\
\propto& \exp\left[g(\boldsymbol{y})^\top \boldsymbol{f} + (\beta(\boldsymbol{y})\circ \boldsymbol{\omega})^\top\boldsymbol{f} -\boldsymbol{f}^\top\mathrm{diag}(\gamma(\boldsymbol{y})\circ \boldsymbol{\omega})\boldsymbol{f} - \frac{1}{2} \boldsymbol{f}^\top K^{-1}\boldsymbol{f}   \right]\\
\propto& \exp\left[\left(g(\boldsymbol{y})+\beta(\boldsymbol{y})\circ \boldsymbol{\omega}\right)^\top \boldsymbol{f}  -\boldsymbol{f}^\top\left[\mathrm{diag}(\gamma(\boldsymbol{y})\circ \boldsymbol{\omega})+\frac{1}{2} K^{-1}\right]\boldsymbol{f}  \right].
\end{align*}
We get immediately a multivariate normal distribution with $-\frac{1}{2} \Sigma^{-1} = -\mathrm{diag}(\gamma(\boldsymbol{y})\circ \boldsymbol{\omega})+\frac{1}{2} K^{-1}$ and $\Sigma^{-1}\mu = g(\boldsymbol{y}) + (\beta(\boldsymbol{y})\circ \boldsymbol{\omega})$. Which corresponds to the result shown in equation (\ref{eq:fullconditionals}).

For the augmented variable $\omega_i$:
\begin{align*}
p(\omega_i|y_i,f_i) \propto& p(y_i|f_i,\omega_i)p(\omega_i)\\
\propto & \exp\left( -\|h(y_i,f_i)\|_2^2\omega_i\right)\pi_\varphi(\omega_i|0)\\
=&\pi_\varphi(\omega_i|\|h(y_i,f_i\|_2).
\end{align*}
Note that the equation~\ref{eq:scale-mixture} gives the normalization constant directly $\varphi(\|h(y_i,f_i)\|_2^2)$ directly.
QED.

\subsection{Computation of the moments and cumulants for the augmentation variable}
\label{appendix:moment}
Given the general class of distribution $\pi_\varphi(\omega|c)$ described in Section~\ref{subsec:augmentation}, moments and cumulants can be easily computed:
The $k$-th moment of a distribution can be computed by taking the $k$-th derivative of the moment generating function (equivalent to  a negative Laplace transform) at $t=0$. For example for the first moment:
\begin{align*}
	\expec{\pi_\varphi(\omega|c)}{\omega} =& \left.\frac{d\laplace{\pi_\varphi(\omega|c)}{-t}}{dt}\right|_{t=0}\\ =&\left.\frac{d}{dt}\left[\mathcal{L}\left[\frac{e^{-c^2\omega}\pi_\varphi(\omega|0)}{\varphi(c^2)}\right](-t)\right]\right|_{t=0}\\
	=& -\frac{1}{\varphi(c^2)}\left.\frac{d}{dt}\left[\mathcal{L}\left[\pi_\varphi(\omega|b,0)\right](t+c^2)\right]\right|_{t=0}\\
	=& \left.-\frac{1}{\varphi(c^2)}\frac{d\varphi\left(t+c^2\right)}{dt}\right|_{t=0}\\
	=& - \left.\frac{d\log\varphi(t)}{dt}\right|_{t=c^2}\\
	=& - \frac{\varphi'(c^2)}{\varphi(c^2)} = \overline{\omega}
\end{align*}

More generally the $k$-th moment $m_k$ is defined as :
\begin{align*}
	m_k =& (-1)^k\frac{1}{\varphi(c^2)}\left.\frac{d^k\varphi(t)}{dt^k}\right|_{c^2}
\end{align*}
And the cumulants $\kappa_k$ are computed using the cumulant generating function (log of the moment generating function)
\begin{align*}
	\kappa_k =& (-1)^k\left.\frac{d^k\log \varphi(t)}{dt^k}\right|_{t=c^2}\\
\end{align*}

\subsection{Algorithm for the sparse case}
\label{appendix:sparse}
\begin{algorithm}
	\caption{Augmented Stochastic Variational Inference}
	\label{alg:sparseVI}
	\begin{algorithmic}

		\STATE {\bfseries Input:} Data $(\mathbf{X},\boldsymbol{y})$, GP model $p(\boldsymbol{y}|\boldsymbol{f},\boldsymbol{u})$, kernel $k$
		\STATE {\bfseries Output:} Approximate posterior {$q(\boldsymbol{u}) =
			\mathcal{N}(\boldsymbol{u}|\boldsymbol{m},\boldsymbol{S})$}
		\STATE Find inducing points inputs $Z$ via $k$-means
		\STATE Compute kernel matrices : $K_{Z}$, $\kappa = K_{XZ}K_Z^{-1}$
		\FOR{iteration $t=1,2,\dots$,}
		\STATE \verb|# Local updates:|
		\STATE Sample minibatch $\mathcal{B} \subseteq \{1,\dots,n\}$
		\FOR{$i \in \mathcal{B}$}
		\STATE $c_i=\sqrt{\expec{q(f)}{h(f_i,y_i)^2}}$
		\STATE $\overline{\omega}_i = \expec{q(\omega_i)}{\omega_i} =  -\varphi'(c_i^2)/\varphi(c_i^2)$
		\ENDFOR
		\STATE \verb|# Natural gradient updates (CAVI):|
		\STATE $\widetilde{\boldsymbol{S}} = \left(\kappa^\top\mathrm{diag}\left(2\overline{\boldsymbol{\omega}} \circ \gamma(\boldsymbol{y})\right)\kappa + K_Z^{-1} \right)^{-1}$
		\STATE $\widetilde{\boldsymbol{m}} = \widetilde{\boldsymbol{S}}\left(K_Z^{-1}\mu_0 + \kappa^\top \left(g(\boldsymbol{y}) + \overline{\boldsymbol{\omega}} \circ \beta(\boldsymbol{y})\right)\right)$
		\STATE $\{\boldsymbol{m}, \boldsymbol{S}\} \leftarrow  (1-\rho^{(t)})\{\boldsymbol{m}, \boldsymbol{S}\} +  \rho^{(t)}\{\widetilde{ \boldsymbol{m}}, \widetilde{\boldsymbol{S}}\}$
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

$\rho^{(t)}$ is an arbitrary learning rate respecting the Robbins-Monroe condition.

\subsection{ELBO Analysis}
\label{appendix:ELBO}
\subsubsection{Full ELBO}
\begin{align*}
\begin{split}\text{ELBO} =& \sum_{i=1}^N\expec{q(f_i,\omega_i)}{\log p(y_i|f_i,\omega_i)}\\
&- \mathrm{KL}[q(f)||p(f)] - \sum_{i=1}^N\mathrm{KL}[q(\omega_i)||p(\omega_i)]
\end{split}
\end{align*}
\begin{align*}
\expec{q}{\log p(y_i|f_i,\omega_i,\theta)} =&\log C(\theta) + g(y_i,\theta)\expec{q(f)}{f} - \expec{q(f)}{h(f_i,y_i)^2}\expec{q(\omega_i)}{\omega_i}\\
=& \log C(\theta) + g(y_i,\theta)m_i-\left(\alpha(y_i)-\beta(y_i)m_i+\gamma(y_i)\left(m_i^2 + S_{ii}\right)\right)\overline{\omega}_i\nonumber\\
\mathrm{KL}[q(f)||p(f)]=& \frac{1}{2}\left[\log \frac{|K|}{|\boldsymbol{S}|}-N+\text{tr}(K^{-1}\boldsymbol{S})+(\boldsymbol{\mu}_0-\boldsymbol{m})^\top K^{-1}(\boldsymbol{\mu}_0-\boldsymbol{m})\right]\nonumber\\
\mathrm{KL}[q(\omega_i)||p(\omega_i)] =& -\expec{q(\omega_i)}{c_i^2\omega_i} - \log \varphi(c_i^2) = -c_i^2\overline{\omega}_i-\log \varphi(c_i^2)\nonumber
\end{align*}
Note that we can take the derivatives of the ELBO and set them to 0 to recover exactly the updates in algorithm \ref{alg:VI}.

\subsubsection{Analysis of the optima}

By setting $c_i^2$ as a function of $\boldsymbol{m}$ and $\boldsymbol{S}$ (and setting $\boldsymbol{\mu}_0$ to 0 for simplicity) we can get an ELBO only depending of the variational parameters of $\boldsymbol{f}$.
\begin{align*}
\text{ELBO}(\boldsymbol{m},\boldsymbol{S}) = C + g^\top\boldsymbol{m} + \frac{1}{2}\left(\underbrace{\log |\boldsymbol{S}| - \text{tr}(K^{-1}\boldsymbol{S})- \boldsymbol{m}^\top K^{-1}\boldsymbol{m}}_{\text{ELBO}_1}\right) + \sum_i\underbrace{\log \varphi (m_i^2+S_{ii})}_{\text{ELBO}_2}\\
\end{align*}

It is easy to show that $\text{ELBO}_1$ is jointly concave in  $\boldsymbol{m}$ and $\boldsymbol{S}$ with a short matrix analysis. However $\text{ELBO}_2$ is more complex : $m_i^2+S_{ii}$ is jointly convex in $\boldsymbol{m}$ and $\boldsymbol{S}$, $\phi(r)$ is by definition convex as well, however $\phi(m_i^2+S_{ii})$ is neither jointly convex or concave in $\boldsymbol{m}$ and $\boldsymbol{S}$.
It is therefore impossible to guarantee that there is a global optima, however the CAVI updates guarantee us a local optima.

\subsubsection{ELBO Gap}

For a fixed $q(f)$ we can compare the ELBO of the original model $\mathcal{L}_{std}(q(f))$ and the augmented model $\mathcal{L}_{aug}(q(f)q(\omega))$. It is then straightforward to compute the difference between the two :

\begin{align*}
\Delta \mathcal{L}=&\mathcal{L}_{std}(q(f)) - \mathcal{L}_{aug}(q(f)q(\omega))\\
=& \expec{q(f)}{\log p(y,f)-\log q(f)-\expec{q(\omega)}{p(y,f,\omega)-\log q(f)q(\omega)}}\\
=& \expec{q(f)q(\omega)}{-\log \frac{p(y,f,\omega)}{p(y,f)} + \log q(\omega)}\\
=& \expec{q(f)q(\omega)}{-\log p(\omega|y,f)+\log q(\omega)}\\
=& \expec{q(\omega)}{\log q(\omega) - \expec{q(f)}{\log p(\omega|y,f)}}\\
=& -c^2\expec{q(\omega)}{\omega}+\expec{q(\omega)}{\log \mathrm{PG}(\omega|1,0)}-\log\varphi(c^2)\\
&+\expec{q(f)}{f^2}\expec{q(\omega)}{\omega}-\expec{q(\omega)}{\log \mathrm{PG}(\omega|1,0)}+\expec{q(f)}{\log\varphi(f^2)}\\
=& -c^2 m - \log \varphi(c^2) + \expec{q(f)}{f^2}m + \expec{q(f)}{\log \varphi(f^2)}\\
\end{align*}
Replacing with the optimal $q^*(\omega)=\frac{e^{-c^2\omega}p(\omega)}{\varphi(c^2)}$ with $c^2=\expec{q(f)}{f^2}$
\begin{align*}
\Delta\mathcal{L}^* =& - \log \varphi(c^2) + \expec{q(f)}{\log \varphi(f^2)}
\end{align*}

\subsubsection{Sparse ELBO}
When using the inducing points approach the ELBO becomes:
\begin{align*}
\begin{split}\text{ELBO} =& \sum_{i=1}^N\expec{q(f_i,u_i,\omega_i)}{\log p(y_i|f_i,u_i,\omega_i)} \\
&- \mathrm{KL}[q(u)||p(u)] - \sum_{i=1}^N\mathrm{KL}[q(\omega_i)||p(\omega_i)]
\end{split}
\end{align*}
\begin{align*}
\expec{q}{\log p(y_i|f_i,\omega_i,\theta)} =&\log C(\theta) + g(y_i,\theta)\expec{q(f,u)}{f} - \expec{q(f,u)}{h(f_i,y_i)^2}\expec{q(\omega_i)}{\omega_i}\\
=& \log C(\theta) + g(y_i,\theta)(\kappa^\top \boldsymbol{m})_i-\left(\alpha(y_i)-\beta(y_i)(\kappa^\top \boldsymbol{m})_i+\gamma(y_i)\left((\kappa^\top\boldsymbol{m})_i^2 + (\kappa^\top \boldsymbol{S}\kappa)_{ii}\right)\right)\overline{\omega}_i\nonumber\\
\mathrm{KL}[q(f)||p(f)]=& \frac{1}{2}\left[\log \frac{|K|}{|\boldsymbol{S}|}-N+\text{tr}(K^{-1}\boldsymbol{S})+(\boldsymbol{\mu}_0-\boldsymbol{m})^\top K^{-1}(\boldsymbol{\mu}_0-\boldsymbol{m})\right]\nonumber\\
\mathrm{KL}[q(\omega_i)||p(\omega_i)] =& -\expec{q(\omega_i)}{c_i^2\omega_i} - \log \varphi(c_i^2) = -c_i^2\overline{\omega}_i-\log \varphi(c_i^2)\nonumber
\end{align*}

\subsection{Proof of equivalence between Jaakkola bound and data augmentation}
\label{appendix:bound}
\citet{jaakkola2000bayesian} proposed an approach purely based on optimization. They are assuming $\log p(y|f)$ contains a part convex in $f^2$: ${\log p(y|f)=\log p_{convex}(f) + \log p_{non-convex}(f,y)}$. Using convexity properties they are creating a bound with a Taylor expansion to the first order around an additional variable $c^2$:
\begin{align*}
\log p_c(f) \geq \log p_c(c) + \frac{d\log p_c(c)}{dc^2}(f^2-c^2)
\end{align*}
Putting it back in the full ELBO, they are now getting a quadratic part in $f$, analytically differentiable, and they just need to optimize the additional variables $\{c_i\}$.
\citet{merkle2014completely} shows that any completely monotone function is log-convex, i.e. $\log \varphi(r)$ is convex.
Therefore we can replace $\log p_c(c)$ by $\log \varphi(r)$ to recover our model in the context of variational inference.
Note that the converse does not hold, therefore the complete monotonicity is a stronger assumption.

\subsection{Likelihoods used for the experiments}
\label{appendix:explikelihood}
We detail all likelihoods used for the experiments and their formulation as in equation (\ref{eq:likelihood}).

\textbf{Laplace Likelihood :}
${\text{Laplace}(y|f,\beta) = \frac{1}{2\beta}\exp\left(-\frac{|f-y|}{\beta}\right)}$\\
\textbf{Logistic Likelihood :} $p(y|f) = \sigma(yf) = \frac{e^{yf/2}}{2\cosh(|f|/2)}$\\
\textbf{Student-T Likelihood :} $p(y|f) = \frac{\Gamma((\nu+1)/2)}{\Gamma(\nu/2)\sqrt{\pi\nu}}\left(1+\frac{(y-f)^2}{\nu}\right)^{-(\nu+1)/2}$\\
\textbf{Matern 3/2 Likelihood :} $p(y|f) = \frac{4\rho}{\sqrt{3}}\left(1+\frac{\sqrt{3(y-f)^2}}{\rho}\right)\exp\left(-\frac{\sqrt{3(y-f)^2}}{\rho}\right)$\\
\begin{center}
\begin{tabular}{l|c|c|c|c|c|c|c}
Likelihood & $C(\theta)$ & $g(y,\theta)$ & $||h(y,f,\theta)^2||_2^2$ & $\alpha(y)$ & $\beta(y)$ & $\gamma(y)$ &$\varphi(r)$ \\\hline
Laplace & $(2\beta)^{-1}$ & $0$ & $(y-f)^2$ & $y^2$ & $2y$ &$1$ & $e^{-\sqrt{r}/\beta}$\\
Logistic & $2^{-1}$ & $y/2$ & $f^2$ & $0$ & $0$ & $1$ & $\cosh^{-1}(\sqrt{r}/2)$\\
Student-T & $\Gamma((\nu+1)/2)/(\Gamma(\nu)\sqrt{\pi\nu})$ & $0$ & $(y-f)^2$ & $y^2$ & $2y$ &$1$ & $(1+\frac{r}{\nu})^{-(\nu+1)/2}$ \\
Matern 3/2 & $4\rho/\sqrt{3}$ & $0$ & $(y-f)^2$ & $y^2$ & $2y$ & $1$ & $(1+\frac{\sqrt{3r}}{\rho})e^{-\sqrt{3r}/\rho}$\\
\end{tabular}
\end{center}
\newpage

\subsection{Extra figures}
\label{appendix:fig}
\subsubsection{Autocorrelation plots}
\setcounter{figure}{3}
\begin{figure}[h]
	\begin{center}
	\includegraphics[width=0.32\textwidth]{plots/part_1/Logistic/lag_plot.png}
	\includegraphics[width=0.32\textwidth]{plots/part_1/StudentT/lag_plot.png}

	\includegraphics[width=0.32\textwidth]{plots/part_1/Laplace/lag_plot.png}
	\includegraphics[width=0.32\textwidth]{plots/part_1/Matern32/lag_plot.png}
	\end{center}
	\caption{Auto-correlation plots for differents with lags from 1 to 10}
\end{figure}
\newpage
\subsubsection{HMC Results}

\begin{table}[h]
	\centering
	\begin{tabular}{|c|ccccc|}
		\hline
		$\epsilon$/$n_{step}$ & & 1 & 2 & 5 & 10 \\
		\hline
		\multirow{3}{30pt}{0.01} & Time/Sample (s)  & 0.037 & 0.045 & 0.077 & 0.133  \\
		 & Lag 1 &  0.999 & 0.993 & 0.978 & 0.963\\
		 & Gelman & 3.14 & 1.02 & 1.00 & 2.05\\
		\hline
		\multirow{3}{30pt}{0.05} & Time/Sample (s)  & 0.036 & 0.046 & 0.080 & 0.12 \\
& Lag 1 & 0.999 & 0.998& \textbf{0.931} & 0.948 \\
& Gelman & 1.72 & 1.18& 1.01 & 3.25 \\
\hline
		\multirow{3}{30pt}{0.1} & Time/Sample (s)  & 0.033 & 0.042 & 0.073 & 0.13 \\
& Lag 1 & 0.997 & 0.996& 0.998 & 0.994\\
& Gelman & 1.11 & 1.04& 1.27& 2.71\\
\hline
	\end{tabular}
	\caption{HMC results for the Laplace likelihood}
\end{table}
\begin{table}[h]
	\centering
	\begin{tabular}{|c|ccccc|}
		\hline
		$\epsilon$/$n_{step}$ & & 1 & 2 & 5 & 10 \\
		\hline
		\multirow{3}{30pt}{0.01} & Time/Sample (s)  & 0.675 & 0.110 & 0.177 & 0.251  \\
		& Lag 1 &  0.999 & 0.999 & 0.997 & 0.993\\
		& Gelman & 3.14 & 1.74 & 1.11 & 1.02\\
		\hline
		\multirow{3}{30pt}{0.05} & Time/Sample (s)  & 0.148 & 0.192 & 0.336 & 0.573 \\
		& Lag 1 & 0.997 & 0.993 & 0.962 & \textbf{0.857} \\
		& Gelman & 1.10 & 1.02& 1.00 & 1.00 \\
		\hline
		\multirow{3}{30pt}{0.1} & Time/Sample (s)  & 0.142 & 0.193 & 0.337 & NA \\
		& Lag 1 & 0.993 & 0.976& 0.864 & NA\\
		& Gelman & 1.03 & 1.01& 1.00 & NA\\
		\hline
	\end{tabular}
	\caption{HMC results for the Student-T likelihood}
\end{table}
\begin{table}[h]
	\centering
	\begin{tabular}{|c|ccccc|}
		\hline
		$\epsilon$/$n_{step}$ & & 1 & 2 & 5 & 10 \\
		\hline
		\multirow{3}{30pt}{0.01} & Time/Sample (s)  & 0.009 & 0.013 & 0.021 & 0.041  \\
		& Lag 1 &  0.999 & 0.999 & 0.998 & 0.994\\
		& Gelman & 3.19 & 1.68 & 1.12 & 1.02\\
		\hline
		\multirow{3}{30pt}{0.05} & Time/Sample (s)  & 0.011 & 0.014 & 0.025 & 0.41 \\
		& Lag 1 & 0.998 & 0.994& 0.968 & 0.871 \\
		& Gelman & 1.11 & 1.03& 1.00 & 1.00 \\
		\hline
		\multirow{3}{30pt}{0.1} & Time/Sample (s)  & 0.011 & 0.014 & 0.024 & 0.048 \\
		& Lag 1 & 0.994 & 0.979& 0.875 & \textbf{0.532}\\
		& Gelman & 1.02 & 1.01& 1.00& 1.00\\
		\hline
	\end{tabular}
	\caption{HMC Results for the Logistic likelihood}
\end{table}
\newpage
\subsubsection{ELBO difference}

\begin{figure}[h]
	\begin{center}
	\includegraphics[width=0.32\columnwidth]{\plotdir{{part_2/ELBO_likelihood=StudentT_variance=0.1}.png}} \includegraphics[width=0.32\columnwidth]{\plotdir{{part_2/NLL_likelihood=StudentT_variance=0.1}.png}}\\
		a)	Student-T likelihood on the Boston Housing dataset\\
	\includegraphics[width=0.32\columnwidth]{\plotdir{{part_2/ELBO_likelihood=Laplace_variance=0.1}.png}} \includegraphics[width=0.32\columnwidth]{\plotdir{{part_2/NLL_likelihood=Laplace_variance=0.1}.png}}\\
		b)	Laplace likelihood on the Boston Housing dataset
\end{center}
	\caption{Converged negative ELBO and averaged negative log-likelihood on a held-out dataset in function of the RBF kernel lengthscale, training VI with and without augmentation.}
\end{figure}
\newpage
\subsubsection{Convergence speed}

\begin{figure}[h]
	\begin{center}
	\includegraphics[width=0.4\textwidth]{plots/part_3/NLL_file_name=HIGGS_likelihood=Logistic_nInducing=200_nMinibatch=100_x_axis=time}
	\includegraphics[width=0.4\textwidth]{plots/part_3/METRIC_file_name=HIGGS_likelihood=Logistic_nInducing=200_nMinibatch=100_x_axis=time}\\
	a) Logistic likelihood on the HIGGS dataset\\
	\includegraphics[width=0.4\textwidth]{plots/part_3/NLL_file_name=airline_likelihood=Matern32_nInducing=200_nMinibatch=100_x_axis=time}
	\includegraphics[width=0.4\textwidth]{plots/part_3/METRIC_file_name=airline_likelihood=Matern32_nInducing=200_nMinibatch=100_x_axis=time}\\
	b) Matern 3/2 likelihood on the Airline dataset\\
	\includegraphics[width=0.4\textwidth]{plots/part_3/NLL_file_name=CASP_likelihood=StudentT_nInducing=200_nMinibatch=100_x_axis=time}
	\includegraphics[width=0.4\textwidth]{plots/part_3/METRIC_file_name=CASP_likelihood=StudentT_nInducing=200_nMinibatch=100_x_axis=time}\\
	c) Student-T likelihood on the Protein dataset
\end{center}
\caption{Supplementary convergence plots}
\end{figure}

\section{Conclusion}
\label{sec:conclusion}
\vspace{-0.4cm}
We proposed a new efficient inference method for GP models that have a super-Gaussian likelihood.
Our method builds on an auxiliary variable augmentation that renders the model conditionally conjugate.
We showed that in the augmented model, variational inference is up to two orders of magnitude faster and more stable than the state of the art.
For small dataset, we proposed a Gibbs sampler that outperforms Hamiltonian Monte Carlo sampling.
Previous methods that build on auxiliary variable augmentations \citep[e.g.][]{wenzel2019efficient} manually derived the augmentation and inference methods, whereas in our approach the whole procedure is fully automated and works for much more general class of models.
Future work may aim on extending our approach to more general models by automatically constructing \emph{hierarchical augmentations} inspired by \citet{galy2019multi} or \citet{donner2018efficient}.

\section{Experiments}
\vspace{-0.2cm}
\label{sec:experiments}
\setlength{\belowcaptionskip}{-10pt}
\setlength{\abovecaptionskip}{0pt}
\setlength{\floatsep}{0pt}
\setcounter{figure}{2}
\begin{figure*}[h]
	\begin{center}
		\includegraphics[width=0.8\textwidth]{figures/ConvPlot_nInducing=200_nMinibatch=100.png}
		\caption{Test negative log-likelihood and test error (classification)/RMSE (regression) as a function of time for different likelihoods.}
		\label{fig:convspeed}
	\end{center}
\end{figure*}

In this section we answer the following questions empirically:
\begin{compactitem}
	\item How does the Gibbs sampling scheme compare to other sampling methods?
	\item What is lost in variational inference by approximating an additional variable?
	\item And what is the gain in speed?
\end{compactitem}
We explore four different cases.
We use three regression models with different likelihood functions: a Laplace likelihood, a Student-t likelihood, a new likelihood inspired by the Matern 3/2 kernel \citep{rasmussen2003gaussian} and one classification model with a logistic likelihood.
All the mathematical details of these augmentations are deferred to the Appendix \ref{appendix:explikelihood}. For the two first experiments we use a full GP without inducing points to have a cleaner analysis of the effect of the augmentation.
For all experiments we use a squared exponential kernel with automatic relevance determination: $k(x,x') = \exp(-\sum_{d=1}^D(x_d-x'_d)^2/\theta_d^2)$.
For the two first experiments we use datasets from the UCI repository \citep{Dua:2019} : the Boston housing dataset ($N=506,D=14$) for regression and the Heart dataset ($N=303,D=14$) for classification.
For the last experiment we use the Protein dataset ($N=45730,D=9$) and the Airline dataset ($N=190K,D=7$) for regression and the Covtype dataset ($N=581K,D=54$) and the SUSY dataset ($N=5$M, $D=18$) for classification.
We normalize the input features to mean 0 and variance 1.
\vspace{-0.3cm}
\subsection{Gibbs sampling mixing}
\label{sec:expgibbs}
Our approach leads to a Gibbs sampling algorithm that provides samples from the true posterior of the original model.
We compare our method (\textit{Gibbs}) with a naive Metropolis-Hasting algorithm (\textit{MH}) and a Hamiltonian Monte Carlo (\textit{HMC}) sampler (where $\epsilon$ and $n_{step}$ are selected via a grid search, see appendix \ref{appendix:fig}) both implemented in Turing.jl \citep{ge2018t}, with a whitening transformation on the kernel matrix for better mixing.
We draw 5 independent chains of 10000 samples for each algorithm.
We compare crucial sampling diagnostics among different models:
we give the autocorrelation between consecutive samples (lag 1) (as well as the autocorrelation plots for all lags in appendix~\ref{appendix:fig}) to estimate the efficient sample size and the chain intercorrelation via the Gelman test (1 is the optimum) \citep{brooks1998general}.
The results are summarized in table \ref{table:sampling}.

\begin{table}
\begin{center}
\begin{tabular}{ccccc}
	\multicolumn{2}{c}{Likelihood/Method} & MH & HMC & \textbf{Gibbs}\\\hline
	\multirow{3}{30pt}{Logistic} & Time/Sample (s) & \textbf{0.001} & 0.041  & 0.01\\
	& Lag 1 & 0.996& 0.53& \textbf{0.11}\\
	& Gelman & 1.38 & \textbf{1.00} & \textbf{1.00}\\\hline
	\multirow{3}{33pt}{\small Student-t}	& Time/Sample (s) & \textbf{0.003} & 0.573 & 0.028 \\
	& Lag 1 & 1.0 & 0.857& \textbf{0.04}\\
	& Gelman & 1.51 & 1.00 & \textbf{1.00}\\\hline
	\multirow{3}{30pt}{Laplace}	& Time/Sample (s) & \textbf{0.002} & 0.082 & 0.028\\
	& Lag 1 & 0.995& 0.931& \textbf{0.26} \\
	& Gelman & 1.44 & 1.01 & \textbf{1.00}\\\hline
	\multirow{3}{30pt}{Matern 3/2}	& Time/Sample (s) & \textbf{0.005} & 0.15 & 0.029\\
	& Lag 1 & 0.997& 0.995& \textbf{0.05}\\
	& Gelman & 1.59 & 1.10 & \textbf{1.00}\\\hline
\end{tabular}
\caption{Sampling time and diagnostics of Gibbs Sampling, naive Metropolis-Hastings and Hamiltonian Monte-Carlo.
The Gelman test indicates the inter-chain correlation and should be close to 1.}
\label{table:sampling}
\end{center}
\end{table}
\setcounter{figure}{1}
\begin{figure}[h]
	\includegraphics[width=0.49\columnwidth]{\plotdir{{part_2/ELBO_likelihood=Laplace_variance=0.1}.png}} \includegraphics[width=0.49\columnwidth]{\plotdir{{part_2/NLL_likelihood=Matern32_variance=0.1}.png}}
	\begin{center}
		a)	Matern 3/2 Likelihood on the Boston Housing dataset
	\end{center}
	\includegraphics[width=0.49\columnwidth]{\plotdir{{part_2/ELBO_likelihood=Logistic_variance=0.1}.png}} \includegraphics[width=0.49\columnwidth]{\plotdir{{part_2/NLL_likelihood=Logistic_variance=0.1}.png}}
	\begin{center}
		b)	Logistic Likelihood on the Heart dataset
	\end{center}
	\caption{Converged negative ELBO and averaged negative log-likelihood on a held-out dataset in function of the kernel lengthscale, training VI with and without augmentation.}
	\label{fig:elbovi}
\end{figure}
We find that our method has a very low intrachain correlation leading to a high sample efficiency, as well as a low interchain correlation while still being faster than the HMC algorithm.
It is even more evident for heavy-tailed likelihood like Student-T or Laplace where HMC can be of more trouble \citep{betancourt2017conceptual}.
Our approach is limited by the $\mathcal{O}(N^3)$ complexity for each sample.

\subsection{Augmentation gap}
\label{sec:auggap}
To investigate the effect of augmenting the model when using variational inference, we train the original model using gradient descent and the augmented model until convergence.
While we fix the kernel variance at 0.1, we vary the lengthscale $\theta$ from $10^{-2}$ to $10^{2}$.
We compare the converged ELBOs as well as the predictive performance on held-out test set.
The results for the matern 3/2 and logistic are shown on figure~\ref{fig:elbovi}, the other likelihoods are show in the appendix~\ref{appendix:fig}.
For both shown likelihoods, there is a visible ELBO gap between the augmented model and the original model.
However the predictive performance is marginally the same for both models
We can conclude that a potential difference in ELBO values does not affect the prediction performance.

\vspace{-0.3cm}

\subsection{Convergence speed}
\label{sec:convspeed}
To scale our model to large datasets, we use the inducing points technique of \citet{titsias2009variational} and we use the stochastic gradient descent approach of \citet{hoffman2013stochastic}.
We compare our variational approach (Algorithm \ref{alg:VI}) to using natural gradient descent, \citep{salimbeni2018natural} and ADAM \citep{hensman2015scalable} both implemented in GPFlow \citep{matthews2017gpflow}.
For all methods we use 200 inducing points determined by $k$-means++ \citep{arthur2007k}, minibatches of size 100 and we train the kernel hyperparameters using ADAM \citep{kingma2014adam}, (the inducing points locations are fixed).
We show the predictive performance in function of the training time for multiple likelihoods on figure \ref{fig:convspeed}.

Our method is up to two orders of magnitude faster than the state of the art.
Moreover, we find that the optimization in our method is more stable (smooth decrease of the loss.
\vspace{-0.3cm}
\section{Introduction}
\label{sec:introduction}

\begin{figure*}[!t]
  \glsresetall
  \centering
  \includegraphics[width=0.9\textwidth]{figures/pipe_line_figure/pipe_line_v2.pdf}
  \caption{\Gls{ACI} performs automated efficient inference in non-conjugate Gaussian process models. In the first step, \gls{ACI} translates the GP model into an augmented model that is conditionally conjugate. In the second step, the complete conditionals are computed in closed form. In the final step, \gls{ACI} provides two options: (A) fast stochastic variational inference based on coordinate ascent updates, which easily scales to big datasets and (B) an asymptotically exact Gibbs sampler, which provides high quality samples from the true posterior but is limited to smaller datasets.}
  \label{fig:pipe_line}
\end{figure*}

\glsresetall
Developing automated yet efficient Bayesian inference methods for Gaussian process (GP) models is a challenging problem that has attracted considerable attention within the probabilisitic machine learning community \citep{salimbeni2018natural, wenzel2019efficient}.
A GP defines a distribution over functions and can be used as a flexible building block to develop expressive probabilistic models.
By choosing an appropriate likelihood function on top of a latent GP, a variety of interesting models is obtained, which are successfully used in several application areas including robotics \citep{GProbot}, facial behavior analysis \citep{eleftheriadis2017tip} and electrical engineering \citep{Pandit_2018}.
For instance, using a logistic likelihood leads to a binary GP classification model, and using a Student-t likelihood can be used for robust regression.

The main challenge in these models is to infer the latent GP given a general non-Gaussian likelihood.
Methods that are more generally applicable often treat the model as a black box and are based on sampling or numerical quadrature, thus, preventing efficient optimization \citep{hensman2015scalable,salimbeni2018natural}.
On the other side. a lot of methods focus on special cases of GP models (i.e. special likelihood functions) by exploiting model specific properties, e.g. binary classification \citep{polson2013bayesian}.

\footnotetext{Work done while at TU Berlin}
\renewcommand*{\thefootnote}{\arabic{footnote}}
\setcounter{footnote}{0}
In this work, we develop \gls{ACI}. \gls{ACI} is an efficient inference framework, which is applicable to a large class of GP models that use a super-Gaussian likelihood\footnote{The definition of the family of super-Gaussian likelihoods is given in Section~\ref{sec:algorithm}.}. It automatically exploits specific properties of the likelihood leading to an inference algorithm that is up to two orders of magnitudes faster than the state of the art

Our approach builds on an auxiliary variable augmentation of the model: we add a latent variable to the model such that the original model is recovered when this variable is integrated out.
We consider an augmentation that renders the model conditionally conjugate. In a conditionally conjugate model, all complete conditional distributions (the posterior distribution of one random variable given all the others), can be computed in closed form. Moreover, we show that inference in the augmented conditionally conjugate model is much easier than in the original model and demonstrate superior performance over the state of the art.

Building on the conditionally conjugate augmentation, \gls{ACI} provides two options for inference: a scalable variational inference method based on efficient closed-form coordinate ascent updates and an exact Gibbs sampling method, which is useful on smaller datasets.

Our main contributions are as follows:
\begin{compactitem}
    \item We introduce \gls{ACI}: an automated inference method for GP models with a super-Gaussian likelihood
    \item We propose two inference modules: augmented variational inference, which scales to large datasets containing millions of instances and an exact Gibbs sampler, which is useful for small datasets.
    \item The experiments demonstrate that the augmented variational inference module of \gls{ACI} outperforms the state of the art in terms of speed by up to two orders of magnitude while being competitive in terms of prediction performance.
    The Gibbs sampler module leads to a much better efficient sample size while still being up to ten times faster than Hamiltonian Monte Carlo.
\end{compactitem}
The paper is structured as follows: Section~\ref{sec:method} gives a high-level overview about our novel inference method \gls{ACI}.
In Section~\ref{sec:algorithm}, we provide a detailed discussion of the algorithm and proof that our approach indeed leads to conditionally conjugate models.
We discuss related work in Section~\ref{sec:related_work} and show our experimental results in Section~\ref{sec:experiments}.
Finally, Section~\ref{sec:conclusion} concludes and lays out future research directions. Our source code for the experiments is included in a gitgub repository\footnote{\url{https://github.com/theogf/AutoConjGP_Exp}}.

\section{Automated Augmented Conjugate Inference}
\label{sec:method}
\glsresetall

Let $X=(\boldsymbol{x}_1, \dots, \boldsymbol{x}_n)^\top \in \mathbb{R}^{n\times d}$ be a matrix of data points and $\boldsymbol{y}=(y_1, \dots, y_n) \in \mathbb{R}^n$ the corresponding target values. The goal is to learn a mapping from the input points to the target values via a latent function $f$. We assume a prior GP distribution (with mean prior $\boldsymbol{\mu}_0$ and covariance function $k(x,x')$) on the latent function and the data labels
 $\boldsymbol{y}=(y_1, \dots, y_n)$ are connected to $f$ via a factorizable likelihood
 \begin{align*}
   p(f) = \mathrm{GP}(f | \boldsymbol{\mu}_0,k),\quad   p(\boldsymbol{y} | f, X) = \prod_{i=1}^n p(y_i | f(\boldsymbol{x}_i)).
 \end{align*}

The key inference challenge in the GP models is to compute the posterior distribution of the latent function
\begin{align*}
    p(f|y) = \frac{p(y|f)p(f)}{\int p(y|f)p(f) \mathrm{d} y}.
\end{align*}
This is a challenging problem.
Inference in GP models scale cubically in the number of data points and is intractable for non-Gaussian likelihoods.

Ideally, we would like an efficient inference method that is not hand-tailored to a specific type of likelihood and hence allows for experimenting with different types of GP models on big datasets in a scalable manner.
Thus, we need a flexible inference method that works for a large class of likelihoods, is fast and ideally does not involve inefficient black box approaches as approximating the objective by sampling.
\subsection{Automated Augmented Conjugate Inference}
We introduce the \gls{ACI} to achieve this goal. \gls{ACI} accelerates
training of GP models whose likelihood is in the family of super-Gaussian likelihood functions.

\gls{ACI} translates the intractable non-conjugate model into an easier, conditionally conjugate model by adding auxiliary random variables to the model.
Inference in conditionally conjugate models is a classic and well-studied problem \citep{bishop2006pattern}.
Because of the special structure of conditionally conjugate models, many efficient inference methods exist \citep{wang2013variational}.
Based on the automatically constructed augmentation, we propose an efficient variational inference method using coordinate ascent updates and a Gibbs sampler.

\parhead{The inference pipeline of \gls{ACI}.}
\gls{ACI} consists of three steps. In the first step, a conjugate augmentation of the model is constructed by adding auxiliary variables $\omega$ to the model. Then, the complete conditional distributions of the latent function $f$ and auxiliary variables $\omega$ are computed.
In the final step, we provide two options to perform inference.

The \emph{variational inference} (VI) module of \gls{ACI} performs block coordinate ascent updates, computed in closed form.
The updates are much more efficient than ordinary Euclidean gradient updates, which are used in most previous approaches.
The \emph{Gibbs sampling} module of \gls{ACI} builds on the complete conditional distributions and provides exact samples from the true posterior.
For each type of likelihood, the sampler is automatically constructed.

The inference pipeline of \gls{ACI} is summarized in Fig.~\ref{fig:pipe_line}. In the following, we give an overview of how each module of our inference pipeline works and provide the details in Section~\ref{sec:algorithm}.

\parhead{(1) Augmenting the model.}
The first step of our inference framework constructs an \emph{auxiliary variable augmentation} that renders the model \emph{conditionally conjugate}. Our augmentation approach finds a Gaussian scale mixture representation of the intractable likelihood
\begin{align}
p(y_i|f_i) = \int p(y_i|f_i,\omega_i) p(\omega_i) \mathrm{d}\omega, \label{eq:scalemixture}
\end{align}
where $p(y_i|f_i,\omega_i)$ is an unnormalized Gaussian distribution in $f_i$ with precision $\omega_i$
and $p(\omega_i)$ is the prior distribution of the auxiliary variable. The construction of the distribution $p(\omega)$ is based on an inverse Laplace transformation and is discussed in Section~\ref{subsec:augmentation}.

Building on Eq.~\ref{eq:scalemixture}, we augment the GP model by a set of auxiliary variables  $\boldsymbol{\omega}=(\omega_1, \dots, \omega_n)$ leading to the augmented joint distribution
\begin{align}
        p(\boldsymbol{y}, \boldsymbol{f}, \boldsymbol{\omega}) = \prod_i p(y_i|f_i, \omega_i) p(\omega_i) p(\boldsymbol{f}), \label{eq:augmented_model}.
\end{align}
The auxiliary variable augmentation is constructed in a way such that the augmented model is \emph{conditionally conjugate}, i.e. the complete conditional distributions $p(\boldsymbol{\omega}|\boldsymbol{f}, \boldsymbol{y})$ and $p(\boldsymbol{f}| \boldsymbol{\omega}, \boldsymbol{y})$ are in the same family as their associated priors.

\parhead{(2) Computing the complete conditionals.}
The complete conditionals of $\boldsymbol{f}$ and the auxiliary variables $\omega_i$ are computed in closed form and are given by
\begin{align*}
\begin{aligned}
  p(\boldsymbol{f}|\boldsymbol{y},\boldsymbol{\omega}) =& \mathcal{N}\left(\boldsymbol{f}| \boldsymbol{\mu} ,\boldsymbol{\Sigma}\right)\\
  p(\omega_i|f_i,y_i) =& \pi_{\varphi}\left(\omega_i| c_i\right),
\end{aligned}
\end{align*}
where $\varphi$ is a function determined by the type of the likelihood (see Eq.~\ref{eq:likelihood}) and the parameters $\boldsymbol{\mu}, \boldsymbol{\Sigma}, c_i$ have closed-form expressions and are described in Section~\ref{subsec:complete_conditionals}.
The distribution family $\pi_{\varphi}\left(\omega| c\right)$ is derived by an exponential tilting
of the prior distribution $p(\omega)$ and is discussed in Section~\ref{subsec:complete_conditionals}.

\parhead{(3a) Augmented variational inference.}
In step 3, \gls{ACI} provides two options to perform inference. We first discuss the variational inference module, which approximates the posterior by a variational distribution and easily scales to big datasets.

We assume a mean-field variational distribution, where the latent GP $\boldsymbol{f}$ and the auxiliary variables $\boldsymbol{\omega}$ are decoupled, i.e. $q(\boldsymbol{f},\boldsymbol{\omega}) = q(\boldsymbol{f})q(\boldsymbol{\omega})$.
The optimal variational distribution of $\boldsymbol{\omega}$ naturally factorizes, i.e. $q(\boldsymbol{\omega}) = \prod_i q(\omega_i)$. Following standard results \citep{bishop2006pattern} the variational distributions can be iteratively optimized by the block-coordinate ascent updates:
\begin{align}
  \begin{split}
    q(\boldsymbol{f})&\propto \exp \left(\expec{q(\boldsymbol{\omega})}{\log p(\boldsymbol{f}|\boldsymbol{\omega},\boldsymbol{y})}\right)\\
    q(\omega_i)&\propto \exp \left( \expec{q(\boldsymbol{f})}{\log p(\omega_i|\boldsymbol{f},\boldsymbol{y})}\right).
  \end{split}
  \label{eq:cavi_updates}
\end{align}
In Section~\ref{subsec:inference}, we show that these updates are given in closed form and can be computed efficiently without resorting to numerical methods.
To scale to big datasets we employ SVI \citep{hoffman2013stochastic} and replace the original latent GP $f$ by \citet{titsias2009variational} sparse approximation building on inducing points .

\parhead{(3b) Exact inference via Gibbs sampling.}
Building on the conditionally conjugate augmentation, it is straightforward to derive a Gibbs sampler. In order to sample from the exact posterior, we alternate between drawing a sample from each complete conditional distribution
\begin{align*}
	\boldsymbol{\omega}^t &\sim p(\boldsymbol{\omega}|\boldsymbol{f}^{t-1}, \boldsymbol{y}),\\
    \boldsymbol{f}^t &\sim p(\boldsymbol{f}|\boldsymbol{\omega}^{t}, \boldsymbol{y}).
\end{align*}
The augmented variables are naturally marginalized out and the latent GP samples $\{\boldsymbol{f}^t\}$ will be from the true posterior $p(\boldsymbol{y}|\boldsymbol{f})$.
As we empirically show in Section~\ref{sec:expgibbs}, the Gibbs sampler leads to very fast mixing and outperforms standard Hamiltonian Monte Carlo sampling.

\section{Related Work}
\label{sec:related_work}
\glsresetall
Inference for non-conjugate likelihoods is not a new topic and there have been many works to deal efficiently with the problem.

\parhead{Scale mixtures of normals.}
The Gaussian scale-mixture formulation is well known in statistics and have been explored more recently by \citet{gneiting1997normal,gneiting1999radial}. \citet{palmer2006variational,articlepalmer2006variational} started to generalize it for a machine learning use but did not explore the probability side of the augmentation.

\parhead{Black-box variational inference.} One of the most popular approach for variational inference in the recent years is to optimize the ELBO for an arbitrary model by computing gradients estimates via sampling or quadrature, e.g. \citet{salimbeni2018natural,mohamed2019monte}.
However these methods do not exploit the structure of the model and can be less efficient.

\parhead{Sampling methods.} Sampling is not a popular method for GP models since $f$ is high-dimensional and the posterior is usually highly correlated \citep{lawrence2009efficient}.
But as for many Bayesian models, Hamiltonian Monte Carlo is a good candidate  \citep{titsias2008markov}.

\parhead{Likelihood approximation.}
\citet{jaakkola2000bayesian} propose a variational approach purely based on optimization, using the partial convexity of the likelihood. Our method recovers their results, but coming from a probabilistic perspective.
We show in Appendix \ref{appendix:bound}, the equivalence with their approach.
\citet{khan2017conjugate} exploit existing partial conjugacy in the model and rely on the assumption that part of the joint posterior can be rewritten as an exponential family.
Their approach is complementary to ours and could be combined for solving more complex models.

\parhead{Use cases of the augmented model.} Different applications of the augmentation technique for specific likelihoods have been explored in multiple papers: \citet{jylanki2011robust} applied the augmentation on the Student-t likelihood with Gaussian Processes. \citet{polson2013bayesian} developed an approach with the logistic likelihood, this work was further expanded by \citet{wenzel2019efficient} to big data.
The augmentation done on the Bayesian Support Vector Machine of \citet{polson2011data} and scaled up by \citet{wenzel2017bayesian}, is similar to our method but is based on a different augmentation approach.
Note that our method covers all these cases exactly but do not rely on any manual derivations.

\section{Additional Stuff}

redpajama_set_name: RedPajamaArXiv


The International Mechanical Code (IMC) and the American Society of Heating Refrigeration, and Air Conditioning (ASHRAE) have established the standardization of proper ventilation in all enclosed parking garage facilities throughout North America. This standardization within a garage building stipulates that these specialized ventilation systems must run continuously throughout the day and night during open business hours. The only exception to this continuous ventilation, is if the building is using carbon monoxide detecting sensors. In this case, the ventilation is not demand controlled by these sensors / controls.
CO sensor based systems are a great alternative to running ventilation systems continuously since they can save the building owner a considerable amount of money over time due to reduce usage of electricity. These CO sensor controls are quite accurate and are typically set to trip the fans or ventilation to ON when they detect 35 parts per million (PPM). Once the contaminated atmosphere has been evacuated from the building, the ventilation will turn OFF automatically, and remain OFF until the CO levels have reached the trip point once again.
Another harmful contaminant that can occur in a sealed buildings atmospheric environment is Radon, which can cause sick building disease.
It is important to have proper air supply and air exhaust, along with air circulation throughout the entire building, as well as within the parking garage itself. When constructing your facility, be sure to choose high quality ventilation equipment. This will reduce down time due to malfunctions and repairs of cheaper quality products.


text:
UFC Sacramento: Germaine de Randamie, Urijah Faber end the night in explosive fashion
Written By E. Spencer Kyte
(Getty Images) https://images.performgroup.com/di/library/sporting_news/91/48/germaine-de-randamie-053017-getty-ftrjpg_bvl0dkhqmfmq1lwvmqkv15oam.jpg?t=1539689340&w=500&quality=80
Aspen Ladd
Germaine de Randamie
The last time the UFC ventured to the Golden 1 Center in Sacramento, Calif., the home of the NBA's Kings was relatively new and Urijah Faber was bidding the sport farewell, locking up with Brad Pickett in his retirement fight in front of a partisan home crowd.
Two and a half years later, the UFC was back in the state capital, and while some things have changed, others remain the same.
Join DAZN and watch more than 100 fight nights a year
The venue is still considered one of the most impressive in The Association, while the Kings have taken major steps forward to become one of the league's most intriguing squads. And just like last time the UFC blew through town, Faber was in the cage, after coming out of retirement a few months after turning 40 and becoming a father.
Here's a look at what transpired Saturday evening and the impact those results will have going forward:
Germaine de Randamie stops Aspen Ladd in 16 seconds
De Randamie showed why she remains a serious threat in the bantamweight division, bouncing Ladd from the ranks of the unbeaten to close out the UFC's return to Sacramento.
As Ladd pressed forward, de Randamie fired a counter right hand that landed flush. Ladd fell to all fours. De Randamie then looked to uncork a follow-up left, but referee Herb Dean stepped in and stopped the fight.
What it means: What it should mean is that de Randamie gets a chance to avenge her lone UFC loss to date in a bantamweight title fight with Amanda Nunes later this year. Both just had quick finishes, no one else is in position to challenge "The Lioness" at 135, and it would be an absolute firefight for as long as it lasts, but . . .
What it probably means is that de Randamie will get booked against Ketlen Vieira once she's cleared to return and the unpopular former featherweight champion will have to keep running the gauntlet to get another championship opportunity after how things went following her featherweight title win two years ago.
It was a rough outing for Ladd, but ultimately, it will probably prove to be beneficial to her long-term development. She'll have time to keep improving, and she didn't catch a bunch of follow-up hammers.
She handled things the right way following the loss and she has a ton of potential, but it's clear there are areas of her game that still need work. There is no reason to hustle her into increasingly dangerous matchups with opponents who are exponentially more experienced.
Urijah Faber returns with a bang, stops Ricky Simon in under a minute
I think most people had the same feeling I did heading into Saturday's co-main event: Faber's walkout is going to be awesome, but the 40-year-old will be stepping in with a surging, young prospect and the night could go really, really poorly.
The walkout was great, as expected, but after getting touched up in one of the first exchanges, Faber hit Simon with an overhand right that took out his feet. From there, "The California Kid" swarmed. Referee Mike Beltran quickly jumped in and halted the action, giving the returning Hall of Famer the fastest finish of his career.
What it means: If I say, "Nothing good," are people going to think I'm a hater?
Here's the thing: It was a terrific moment for Faber - to come out of retirement and score a win like that at home was electric - but now what? Henry Cejudo called him out by name six weeks ago, Faber said after his victory that he was down, and the rest of the division surely let out a collective groan because it could happen, even though it has no business happening.
When I spoke with Faber before the bout, he said he was interested in big fights, and that is surely a big fight, but even he wasn't talking about a title fight at the time. It's a possibility because, of course it is, but it's one of those contests that will only cause more chaos if it's booked.
This is a rough result for Simon, who had won three straight to build momentum and creep into the bantamweight rankings. It's not the end of the world by any stretch and no one would have limited the bump he would have gotten had he been victorious, but this feels like a bit of a wasted loss for a talented, emerging fighter. Faber is closer to being a mercenary than a permanent member of the roster.
Josh Emmett blasts Mirsad Bektic, remains unbeaten at home
Emmett and Bektic spent the first three minutes and change standing in front of each other, slinging heavy leather with designs on creating an opportunity to finish. Late in the opening frame, Emmett stuck Bektic with a stiff jab; from there, he pounced, pounding out the finish to remain undefeated while fighting in Sacramento.
What it means: Not a lot of people were talking about this fight going in, but it was a crucial one in the featherweight division and a massive performance for the 34-year-old Emmett, who is now 4-1 in the UFC at 145 pounds and the owner of back-to-back stoppage wins.
Emmett felt he was close to contention prior to his crushing loss to Jeremy Stephens in February 2018, and his last two victories should carry him back into a similar position. He's one of the more powerful fighters in the division and has wrestling skills to fall back on if needed, which makes him a tough out for anyone in the top 10.
Featherweight is heating up and tons of quality matchups that could work for Emmett going forward, including bouts against Yair Rodriguez, Chan Sung Jung or Zabit Magomedsharipov.
As for Bektic, he's now 0-2 against Team Alpha Male and undefeated against everyone else. He was more than holding his own through the early going and he remains a top-15 fighter in the division, but he once again has to hit the reset button and find a way to close the holes in his striking defense to make another push toward the top of the division.
Karl Roberson squeaks out split-decision victory over newcomer Wellington Turman
Roberson was originally scheduled to face John Phillips, but an injury sidelined the Welshman and opened the door for the 22-year-old Turman, who came ever so close to walking away from his UFC debut with a victory.
After a strong start on the feet, Roberson got sucked into a grappling match with Turman. At various points, the Brazilian advanced to dominant positions and threatened with submissions, forcing Roberson to defend. While he was unable to secure the finish, it seemed as though Turman was on his way to stealing the win, but Roberson reversed positions and popped the debutant with a big right hand just before the horn.
What it means: Honestly, the biggest takeaway from this fight was that Turman has solid upside and is someone to keep an eye on. Roberson may have gotten the victory, but it was ultra-close and you have to wonder how things would have played out if Turman had more than a month to prepare for his UFC debut.
Roberson is a dangerous, explosive striker, but he has yet to figure out how to just play to his strengths and avoid troublesome spots, something that nearly cost him Saturday. He'll need to sort that out if he wants to make headway in the division. Turman, meanwhile, should hold his head high after this one.
Marvin Vettori returns to outwork Cezar Ferreira
Fighting for the first time in more than 400 days, Vettori snapped a two-fight winless skid, registering his first victory in two years by outworking former "TUF: Brazil" winner in the main-card opener.
The 25-year-old Italian, who went the distance in a split-decision loss to current interim champ Israel Adesanya last time out, started quickly and maintained his lead throughout, busting up Ferreira's nose in the first before continuing to touch up "Mutante" throughout.
What it means: This is a crucial victory for Vettori, who has always been considered a quality prospect but who seemed to stall out after a very good start to his UFC career. Ferreira may not be a contender but he has beaten some talented opponents, and Vettori never gave him a chance to find his footing and mount any real offense.
"The Italian Dream" is brimming with confidence and is just entering his athletic prime, so his best days should, in theory, be ahead of him. He showed strong takedown defense on Saturday and clean, crisp hands, though his gas tank continues to be a concern as he slowed noticeably down the stretch.
He called for a fight with Paulo Costa, but "Borrachina" is otherwise engaged and, quite frankly, out of his league at this point. That said, you have to appreciate the moxie, and a performance like this one should elevate him toward the top 15 and a tougher matchup later this year.
UFC Sacramento official results
Germaine de Randamie def. Aspen Ladd by KO (right hand) at 0:16 of Round 1
Urijah Faber def. Ricky Simon by KO (strikes) at 0:46 of Round 1
Josh Emmett def. Mirsad Bektic by TKO (strikes) at 4:25 of Round 1
Karl Roberson def. Wellington Turman by split decision (29-28, 28-29, 29-28)
Marvin Vettori def. Cezar Ferreira by unanimous decision (30-27, 30-27, 30-27)
John Allan def. Mike Rodriguez by unanimous decision (29-28, 29-28, 30-27)
Andre Fili def. Sheymon Moraes by TKO (strikes) at 3:07 of Round 1
Juliana Pena def. Nicco Montano by unanimous decision (29-28, 29-27, 29-28)
Ryan Hall def. Darren Elkins by unanimous decision (29-28, 29-28, 30-27)
Jonathan Martinez def. Pingyuan Liu by KO (knee) at 3:54 of Round 3
Brianna Van Buren def. Livia Renata Souza by unanimous decision (30-27, 30-27, 30-27)
Benito Lopez def. Vince Morales by unanimous decision (29-28, 29-28, 29-28)


text:
Home » Racism and Self-Determination » Detroit MLK Day rally highlights Selma, Ala.
Detroit MLK Day rally highlights Selma, Ala.
By Abayomi Azikiwe posted on January 24, 2015
MLK Day march in Detroit on Jan. 19.
WW photo: Abayomi Azikiwe
This year's "Dr. Martin Luther King Jr. Rally and March" commemorated the 50th anniversary of the Selma campaign that created the conditions for the passage of the Voting Rights Act of 1965. The program was
organized by the Detroit MLK Committee, a coalition of community
organizations and churches involved in the ongoing movements for human rights, peace and self-determination.
On Jan. 19, a standing-room-only crowd gathered in the sanctuary of the Central United Methodist Church, where Dr. King had spoken on numerous occasions between 1958 and 1968. Just three weeks prior to his assassination, on March 14, 1968, he addressed the city of Detroit at his annual Lent sermon delivered for the last time.
The 2015 gathering was one of the largest since 2004, when the event was founded, organized by the Michigan Emergency Committee Against War & Injustice (MECAWI) during the first year. In 2005, a broader alliance was formed which has continued to plan the gathering since then.
The meeting was called with the theme "50 Years Since Selma: The Struggle for Democracy, Peace and Social Justice Continues." A program introduction read, "We are recognizing the valiant contributions of the struggle for voting rights by Dr. King … and others who paved the way for the advances gained during the 1960s and 1970s.
"Nonetheless, today we are facing the most profound challenge to the status of civil rights, human rights and economic justice since the martyrdom of Dr. King in 1968. In the state of Michigan, fundamental rights to a living wage, collective bargaining, municipal pensions, public services and basic democratic rights have been eroded."
Community speaks, activists honored
The new pastor at Central United Methodist Church, the Rev. Dr. Jill Hardt Zundel, welcomed people, emphasizing the long tradition of support for social justice movements, the working class and the poor. The church provides regular meals, clothing and other services to homeless people living in the downtown area.
The program paid a special tribute to the late Judge Claudia House Morcom, who died last August, with talks by Dr. Gloria Aneb House, her niece by marriage; Cheryl LaBash of the National Network on Cuba, which Judge Morcom worked with to free the now-released Cuban Five as well as to ease the embargo against Cuba; and Carol Lane, a supporter of Cuba and a veteran peace activist in Detroit. Judge Morcom had been a major supporter of Detroit MLK Day, attending every year.
Cecily McClellan of the Detroit Active and Retired Employees Association (DAREA), which is appealing the theft of pension funds and health care programs from former municipal workers, acknowledged Morcom's pioneering legal work during the Civil Rights Movement and decades-long political activism. McClellan stressed that the imposition of emergency management and bankruptcy was merely a cover to loot over $6 billion in pension funds in the city of Detroit.
Frank Hammer and Melvin Thompson, past presidents of UAW Locals 909 and 140A, read solidarity statements in support of injured Colombian GM autoworkers. With GM world headquarters located just six blocks away from the rally, Thompson and Hammer reflected on the important role of workers in the United States in demanding that industry bosses fulfill their obligations to provide assistance to the injured Colombian workers.
Councilwoman Mary Sheffield presented official city of Detroit recognition awards to Dr. Silas Norman, the associate dean of Admissions at the Wayne State University Medical School, who had served as director of the SNCC voting registration campaign in Selma between 1963 and 1965; Martha Prescod Noonan, a veteran SNCC field secretary in Alabama and Mississippi, an educator and co-editor of the seminal "Hands on the Freedom Plow" book published four years ago by the women of SNCC; and the Rev. Edith Woolen-Worthy, the chairperson of the Social Justice Committee at Hartford Memorial Baptist Church.
Peoples' attorney Alice Jennings addressed the ongoing struggle for water rights in Detroit. Jennings was the lead counsel in the class-action lawsuit filed in bankruptcy court during July 2014, demanding a moratorium on water shutoffs and the adoption of a genuine affordability plan for residents. Jennings has filed an appeal in federal court after bankruptcy Judge Steven Rhodes refused to issue an order to halt the termination of services to the working people and poor of Detroit.
Also, attorney Jerry Goldberg of the Moratorium NOW! Coalition spoke to the need to continue the struggle against the banks. Goldberg intervened in the city's bankruptcy on behalf of retired city employee David Sole, and by doing so, was able to put emergency management and the financial institutions on trial.
Goldberg conveyed that the city is facing yet another burgeoning crisis, with 62,000 property tax foreclosures looming while the right-wing state government refuses to utilize hundreds of millions of dollars of federal housing assistance funds that could pay all delinquent residential tax bills and therefore avert a major crisis of evictions and displacement. He called for an immediate moratorium on all foreclosures in Detroit.
The meeting also focused on the Rev. Edward Pinkney, a political prisoner in Michigan from Berrien County in the southwest region of the state. An all-white jury found him guilty of unproven felony forgery charges related to a recall campaign against the mayor of Benton Harbor. Marcina Cole, an advocate for Rev. Pinkney, called on people to demand that the activist be released pending an appeal of his unjust convictions. Rev. Pinkney was sentenced to 30-120 months in prison and is now incarcerated in Marquette Prison, some eight hours away from his home.
Other speakers included Marian Kramer, who paid tribute to the late labor organizer Gen. Gordon Baker Jr. Pat Driscoll, a retired United Steelworkers union member, read a statement of solidarity with the Boston school bus drivers, represented by USW Local 8751. Four of Local 8751's leaders remain terminated by Veolia Corp. after an illegal lockout during the fall of 2013.
Music during the rally was provided by the Deep River Choir, under the
direction of Ms. Bobbi Thompson. The rally ended with a send-off by the Deep River Choir, singing "This Little Light of Mine," a freedom song utilized by the Selma campaign.
Taking to the streets: 'Black Lives Matter'
The march through downtown was led by African-American youth who
carried a "Black Lives Matter" banner. Additional banners and signs read "Jail Killer Cops," "Stop the Theft of Our Pensions" and "End Poverty, Racism and War." Placards featured photographs of Michael Brown, Aiyana Stanley Jones, Eric Garner and many other victims of police violence.
This demonstration went past the 36th District Court to emphasize the
crisis of foreclosures and evictions in Detroit. The court presides over thousands of evictions every year, many of which are illegal.
Then, the crowd marched on the Wayne County Jail to illustrate the crisis of mass incarceration of African-American and Latino/a youth. The demonstration continued, proceeding through the Greek Town entertainment and casino district, where people came onto the streets to watch and photograph the marchers, who extended back several blocks.
After this, the march went to the headquarters of the Detroit Water and
Sewerage Department, demanding a moratorium on shutoffs, protesting
the regionalization and attempted privatization of the publicly owned system, one of the largest in the U.S. Heading back to the church, demonstrators chanted "No justice! No peace!" "Black lives matter!" "I can't breathe!" and other slogans.
Back at the church, a community meal had been prepared by the "Detroit
Wobbly Kitchen" and "Food Not Bombs" organizations. These groups have provided this meal for the last eleven demonstrations.
Then, a cultural program highlighting local artists was held, coordinated by Detroit poet and teacher Aurora Harris of Broadside Press.
The event is financed solely by community organizations, concerned religious groups and supportive individuals. This rally and march represents one of the largest annual gatherings of progressive forces in southeast Michigan.
Detroit Water
SNCC
Ukraine: U.S./NATO-backed fascist coup & a growing people's resistance


text:
from django.apps import AppConfig

class InventoryInputConfig(AppConfig):
    name = 'inventory_input'

redpajama_set_name: RedPajamaGithub


swisspeace conducted a mediation and negotiation training course in Kathmandu in the framework of the Mediation Support Project, a joint venture with the Center for Security Studies (CSS) of the Swiss Federal Institute of Technology Zurich, in collaboration with the Mediation Support Unit (MSU) of the United Nations and Conflict Management International (CMI). A total of 12 staff members of the Political Division of the UN Mission in Nepal (UNMIN) and 22 military observers participated in the one-week course from 22-28 February 2010.
22 military observers participated in the one-week course on mediation and negotiation skills.
A tailor-made, interactive program with theoretical inputs, exercises and role playing was prepared for each group, adapted to their specific needs. In the case of the members of the UNMIN Political Division, the role playing was as UN representatives mediating between the police and IDPs. As for the military observers these were placed in a fictive negotiating situation, in which they had to mediate between demobilized combatants of a rebel group – who had left their camp armed to provide security for a commander – and the local police. The training team also offered a special one-day coaching program for the head of the UNMIN mission, prepared after consultation to meet her specific requirements.
The Political Affairs Officers used the mediation techniques during a role play, where they were mediating between the police and Internally Displaced Persons.
The training team consisted of one person each from swisspeace and the MSU, an expert in security issues relating to peace processes from the MSU expert standby team and an experienced mediator from the UK. The course and the practical role playing proved popular with the participants. What they learned is to some extent already being put to use in their daily work. Many expressed the desire for regular training in negotiation and mediation skills in the future.


Whole Duch Foie Gras mi-cuit 40g. Castaing. 12un.
Whole Duch Foie Gras mi-cuit 135g. Castaing. 12un.
Whole Goose Foie Gras mi-cuit 135g. Castaing. 12un.
Block of Duck Foie Gras 135g. Castaing. 12un.
Block of Goose Foie Gras 135g. Castaing. 12un.
Foie entero 1,6kg. El Graner.


Freezing embryos is used when extra embryos are obtained after ICSI, so they can be used and implanted later when the couple decide.
It is used in case of cancer or in case of oligospermia so sperm can be preserved for ICSI use.
In case of cancer or early infertility signs, a layer of the ovary is cut and frozen so it can be returned back to the uterus when the female wants to have children in the normal way.
It is used to preserve the eggs for later pregnancies for ICSI use.


Posted by on Monday, May 5th, 2008 at 1:48pm.
You may have already heard the news that Prudential will be changing its name to Carolina One Real Estate. We're excited about this change and wanted to pass the news along to you, our website users! The company opted to not renew its ten-year franchise agreement with Prudential coming up this year. This change was effective starting May 1. We're still in the process of changing everything, so please bear with us during this time of transition!
Everything will still be operated the same way (and by the same people) as before. So, we're still locally owned and operated, but we'll now be affiliated with a larger, global network of real estate companies. And, now we'll have more flexibility and will be able to enjoy the freedom of being an independent company. We've included below the official letter that just appeared in a full page of the Post & Courier.
Today, real estate in Charleston is changing! We are excited to announce that we are changing our name from Prudential Carolina Real Estate to Carolina One Real Estate, and affiliating with Leading Real Estate Companies of the World. The flexibility of a strong local brand tied to a global network will allow us to best serve the real estate needs of the Greater Charleston community.
Leading Real Estate Companies of the World is a network of over 700 independent companies in 38 countries that does more real estate business than any franchise worldwide. Some Leading Real Estate Companies of the World members whose names you may recognize include Allen Tate, # 1 in North Carolina, and Long and Foster, #1 in the Virginia/Maryland area.
Carolina One Real Estate has done business in Charleston under several different names over our 45 year history. Each decision to change was always based on what was in the best interest of our clients. As technology and consumer behaviors change, so must we. The impact of the Internet and alternative advertising trends, including the need for a global connection, especially for our luxury market, make us confident that the decision to become independent of a franchise is the right decision, both today and going forward. It allows us more flexibility to respond to local market needs; directs more of our marketing funds locally; and our new network immediately increases national and international exposure for all of our listings. What remains unchanged is the same continuing local ownership and leadership; the fact that we are the market leader, selling 3 of every 10 homes sold in Charleston; and our commitment to you.
Look for our new signs going up in your neighborhood over the next few days. If you have questions – about our new name or just about real estate in general – please contact your Carolina One Real Estate associate. As always, it is our privilege to earn your trust, one sale at a time.


The enigmatic MYC "central portion". MYC family members are among the most intensely studies proteins in human history. Structure-function analyses have delineated critical activation and DNA-binding domains in the two extremes of MYC, but the central portion that lies between these domains is largely uncharacterized. We believe that understanding the function of this region can reveal novel MYC interaction partners and mechanisms of action that can be exploited to block MYC function in cancer cells.
The WDR5–MYC nexus. Our interest in the MYC central portion led to identification of the WD40-repeat-containing protein WDR5 as a direct MYC binding partner. We hypothesize that WDR5 facilitates the recruitment of MYC to chromatin. We also hypothesize that properties of the MYC–WDR5 interface make it a viable point for discovery of small molecule inhibitors of MYC activity. We want to know how MYC and WDR5 work together, what other proteins are involved, and how understanding of the MYC–WDR5 nexus can inform anti-cancer therapies.


Crochet Glove II Junior a very popular glove with a soft leather palm for good grip when riding.
These Mountain Horse gloves are a very popular riding glove. Crochet Glove II Junior have a soft leather palm which gives good grip. There are added reinforcements between the fingers for strength and grip when holding the reins.
The cotton crochet upper keeps your hands cool and there is an elasticated band at the wrist to keep the glove secure. An attractive horse shoe design features on the back with Mountain Horse logo.