text:
\section{Conclusions}
\label{sec:conclusions}

We proposed a novel algorithm for discretizing latent representations in trained latent variables, with applications to both data and model compression.
Our proposed ``Variational Bayesian Quantization'' (VBQ) algorithm automatically adapts encoding accuracy to posterior uncertainty estimates, and is applicable to arbitrary probabilistic generative models with factorized prior and mean-field variational posterior distributions.
As our approach separates quantization from model design and training, it enables ``plug-\&-play'' compression at variable bitrate with pretrained models.

We showed the empirical effectiveness of our proposed VBQ method for both model and data compression.
For model compression, VBQ retained significantly higher task performance of a trained word embedding model than other  methods that compress in post-processing.
For data compression, we showed that VBQ can outperform JPEG over a wide range of bitrates with a single trained standard VAE.
Given its versatility, we believe that VBQ holds promise for compressing Bayesian neural networks, especially in applications that demand rate-distortion trade-offs.
 Lastly, as VBQ relies on
 accurate posterior approximation
 , its rate-distortion performance provides a new metric for quantitative evaluation of approximate Bayesian inference methods.

\section{Related Work}
\label{sec:related}

Compressing continuous-valued data is a classical problem in the signal processing community.
Typically, a distortion measure (often the squared error) and a source distribution are assumed, and the goal is to design a quantizer that optimizes the rate-distortion (R-D) performance \citep{lloyd1982least, Berger1972OptimumQA, Chou1989EntropyconstrainedVQ}.
Optimal vector quantization, although theoretically well-motivated \citep{gallager1968information}, is not tractable in high-dimensional spaces \citep{gersho2012vector} and not scalable in practice.
Therefore most classical lossy compression algorithms map data to a suitably designed semantic representation, in such a way that coordinate-wise scalar quantization can be fruitfully applied.

Recent machine-learning-based data compression methods learn such hand-designed representation from data, but similar to classical methods, most such ML methods directly take quantization into account in the generative model design or training.
Various approaches have been proposed to approximate the non-differentiable quantization operation during training, such as stochastic binarization \citep{toderici2015variable,toderici2017full}, additive uniform noise \citep{balle2017end,balle2018variational,habibian2019video}, or other differentiable approximation \citep{agustsson2017soft, theis2017lossy, mentzer2018conditional, rippel2017realtime}; many such schemes result in quantization with a uniformly-spaced grid, with the exception of \citep{agustsson2017soft}, which optimizes for quantization grid points. \citet{yang2020improving} considers optimal quantization at compression time, but assumes a fixed quantization scheme of \citep{balle2017end} during training.

We depart from such approaches by treating quantization as a post-processing step decoupled from model design and training.
Crucial to our approach is
a new quantization scheme that automatically adapts to different length scales in the representation space based on posterior uncertainty estimates.
To our best knowledge,
the only prior work that uses posterior uncertainty for compression is in the context of bits-back coding \citep{honkela2004variational, townsend2019practical}, but these works focus on lossless compression, with the recent exception of \citep{yang2020improving}. %

Most existing neural image compression methods require training a separate machine learning model for each desired bitrate setting \citep{balle2017end, balle2018variational, mentzer2018conditional,theis2017lossy,lombardo2019deep}.
In fact, \citet{alemi2018fixing} showed that any particular fitted VAE model only targets one specific point on the rate-distortion curve.
Our approach has the same benefit of variable-bitrate single-model compression as methods based on recurrent VAEs \citep{gregor2016conceptual, toderici2015variable, toderici2017full, johnston2018improved}; but unlike these methods, which use dedicated model architecture for progressive image reconstruction, we instead focus more broadly on quantizing latent representations in a given generative model, designed and trained for specific application purposes (possibly other than compression, e.g., modeling complex scientific observations).

\section{Theoretical Considerations}
\label{sec:theory}
Here, we provide additional theoretical insights into the proposed VBQ method based on reviewer feedback.

\paragraph{Dense Quantization Grid.}
Section~\ref{sec:bayesian-ac} describes the proposed VBQ algorithm in terms of quantiles~$\xi_i$.
While quantiles simplify the discussion, it is also instructive to consider what effectively happens directly in representation space.

\begin{figure}[t]
    \vskip 0.06in
    \centering
    \includegraphics[width=0.49\textwidth]{figs/adaptive-coding/grids}
    \vspace{-0.2in}
    \caption{Dense quantization grid of VBQ (left) and a subset of it (center) that resembles the more common uniform grid (right).}
    \label{fig:grids}
\end{figure}

In the space of representations~${\mathbf z}$, VBQ optimizes over a dense quantization grid, shown in Figure~\ref{fig:grids} (left) for a single dimension~$z_i$.
Each code point $\hat{\xi_i} \in (0, 1)$, i.e., each quantile with a finite-length binary representation, defines a grid point $\hat{z_i} = F^{-1}(\hat \xi_i)$.
The height of each orange bar in the figure shows the bitlength $\mathcal R(\hat z_i)$.
Interestingly, the grid places many points with small $\mathcal R(\hat z_i)$ in regions of high prior probability density (purple curve), while regions of low prior probability contain only grid points with large $\mathcal R(\hat z_i)$.
This observation can be formalized as follows.

\begin{theorem} \label{thm:almost-equidistant}
    For a latent dimension~$z_i$ with prior $p(z_i)$ and for any interval $\mathcal I\subset \mathbb R$, there exists a grid point $\hat z_i \in \mathcal I$ whose bitlength is bounded by the information content of~$\mathcal I$, i.e.,
    $\mathcal R(\hat z_i) \leq - \log_2 P(\mathcal I)$ with $P(\mathcal I) = \int_{\mathcal I} p(z_i)\, dz_i$.
\end{theorem}
\begin{proof}
    $\mathcal R(\hat z_i)$ is the number of nontrivial bits in the quantile $\hat\xi_i := F(\hat z_i)$.
    For example, the quantiles $\frac14={(0.01)}_2$, $\frac12={(0.1)}_2$, and $\frac34={(0.11)}_2$ each have at most one nontrivial bit (since the initial~``$0.$'' and terminal~``$1$'' are considered trivial), and they are equally spaced with spacing $\frac14 = 2^{-2}$.
    Incrementing the number of bits by one divides the spacing in half.
    Thus, for any $r\in\mathbb N$, the quantiles with at most~$r$ nontrivial bits are equally spaced with spacing $2^{-r-1}$.
    The prior CDF~$F$ maps any interval $\mathcal I\subset \mathbb R$ to an interval $F(\mathcal I)\subset(0,1)$ of size $|F(\mathcal I)| = P(\mathcal I)$.
    Since $P(\mathcal I) > 2^{-r-1}$ for $r=\lfloor -\log_2 P(\mathcal I)\rfloor$, the interval~$\mathcal I$ contains at least one grid point $\hat z_i$ with $\mathcal R(\hat z_i) \leq r$.
\end{proof}

\paragraph{Delimination Overhead.}
The above theorem provides an upper bound on the bitrate for a single coordinate~$z_i$.
Encoding a high dimensional latent representation~${\mathbf z}$
requires some overhead for delimiting the individual coordinates.

We provide a theoretical upper bound on this overhead for a severely restricted variant of VBQ that does not have access to posterior uncertainty estimates and that operates only on a subset of the proposed grid.
Specifically, we pick only one grid point for each interval $\mathcal I_n = [n-\frac12, n+\frac12)$ with $n\in\mathbb Z$.
According to Theorem~\ref{thm:almost-equidistant}, there always exists a grid point $\hat z_i \in \mathcal I_n$ with $\mathcal R(\hat z_i) \leq -\log_2 P(\mathcal I_n)$.
Thus, the resulting subset of the grid (Figure~\ref{fig:grids}, center) resembles the more commonly used uniform grid (Figure~\ref{fig:grids} right), whose bitrate under standard entropy coding is the information content.

If we restrict VBQ to this sparse subset of the dense grid, then the algorithm collapses to a method known as Shannon-Fano-Elias coding \citep{cover2012elements}, whose overhead over the information theoretically minimal bitrate is known to be at most one bit per dimension.
The full VBQ algorithm has more freedom: it exploits posterior uncertainty estimates to reduce accuracy where it is not needed, thus saving bits and improving compression performance.

\section{Experiments}
\label{sec:experiments}

We tested our approach on two very different domains: word embeddings and images.
For word embeddings, we measured the performance drop on a semantic reasoning task due to lossy compression.
Our proposed VBQ method significantly improves model performance over uniform discretization and compression with either Arithmetic Coding (AC), gzip, bzip2, or lzma at equal bitrate.
For image compression, we show that a single standard VAE, compressed with VBQ,  outperforms JPEG and other baselines at a wide range of bitrates, both quantitatively and visually.

\subsection{Compressing Word Embeddings}
\label{sec:word-embeddings}

\begin{figure}[t]
\centering
 \includegraphics[width=\columnwidth]{figs/word-embeddings.pdf}
\caption{Performance of compressed word embeddings on a standard semantic and syntactic reasoning task~\cite{mikolov2013efficient}.
VBQ (orange, proposed) leads to much smaller file sizes at equal model performance over a wide range of performances.}
\label{fig:word-embeddings}
\end{figure}

We consider the Bayesian Skip-gram model for neural word embeddings~\citep{barkan2017bayesian}, a probabilistic generative formulation of word2vec~\cite{mikolov2013distributed} which interprets word and context embedding vectors as latent variables and associates them with Gaussian approximate posterior distributions. Point estimating the latent variables would result in classical word2vec.
Even though the model was not specifically designed or trained with model compression taken into consideration, the proposed algorithm can successfully compress it in post-processing.

\paragraph{Experiment Setup.}
We implemented the Black Box VI version of the Bayesian Skip-gram model proposed in \citep{bamler2017dynamic},%
\footnote{See Supplementary Material for hyperparameters. Our code is available at \url{https://github.com/mandt-lab/vbq}.}
and trained the model on books published between $1980$ and $2008$ from the Google Books corpus~\citep{michel2011quantitative}, following the preprocessing described in \citep{bamler2017dynamic} with a vocabulary of $V=100{,}000$ words and embedding dimension $d=100$.

In the trained model, we observed that the distribution of posterior modes $\mu_{w,j}$ across all words~$w$ and all dimensions~$j$ of the embedding space was quite different from the prior.
To improve the bitrate of our method, we used an ``empirical prior'' for encoding that is shared across all~$w$ and~$j$; we chose a Gaussian $\mathcal N(0,\sigma_0^2)$ where $\sigma_0^2$ is the empirical variance of all variational means $(\mu_{w,j})_{w=1,\ldots, V;\, j=1,\ldots, d}$.

We compare our method's performance to a baseline that quantizes to a uniform grid and then uses the empirical distribution of quantized coordinates for lossless entropy coding.
We also compare to uniform quantization baselines that replace the entropy coding step with the standard compression libraries gzip, bzip2, and lzma.
These methods are not restricted by a factorized distribution of code points and could therefore detect and exploit correlations between quantized code points across words or dimensions.

\begin{figure}[t!]
\begin{center}
    \includegraphics[width=\columnwidth]{figs/img_compression/mnist-frey.pdf}
\end{center}
\caption{Qualitative behavior of our proposed VBQ algorithm on two data sets of small-scale images (MNIST and Frey Faces).
With decreasing bitrate, the method starts to confuse the encoded object with a generic one (encoded by the median of the prior $p({\mathbf z})$).
}
\label{fig:mnist-series}
\end{figure}

We evaluate performance on the semantic and syntactic reasoning task proposed in \citep{mikolov2013efficient}, a popular dataset of semantic relations like ``Japan~:~yen = Russia~:~ruble'' and syntactic relations like ``amazing~:~amazingly = lucky~:~luckily'', where the goal is to predict the last word given the first three words.
We report Hits@10, i.e., the fraction of challenges for which the compressed model ranks the correct prediction among the top ten.

\paragraph{Results.}
Figure~\ref{fig:word-embeddings} shows the model performance on the semantic and syntactic reasoning tasks as a function of compression rate.
Our proposed VBQ significantly outperforms all baselines and reaches the same Hits@10 at less than half the bitrate over a wide range.%
\footnote{
The uncompressed model performance (dotted gray line in Figure~\ref{fig:word-embeddings}) is not state of the art.
This is not a shortcoming of the compression method but merely of the model, and can be attributed to the smaller vocabulary and training set used compared to \citep{mikolov2013distributed} due to hardware constraints.}

\subsection{Image Compression}
\label{sec:image-compression}

\begin{figure*}[t!]
\centering
    \begin{subfigure}[h]{0.19\textwidth}
    {\includegraphics[width=\textwidth]{figs/img_compression/reconstructions/flowers.png}}\caption{Original \newline}
    \end{subfigure}
    \begin{subfigure}[h]{0.19\textwidth}
    {\includegraphics[width=\textwidth]{figs/img_compression/reconstructions/flowers.png-jpg-quality=4-BPP=0.245-PSNR=21.371-MSSSIM=0.813.png}}\caption{JPEG \newline \hspace*{1em} MS-SSIM=$0.813$}
    \end{subfigure}
    \begin{subfigure}[h]{0.19\textwidth}
    {\includegraphics[width=\textwidth]{figs/img_compression/reconstructions/flowers.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.244-PSNR=25.880-MSSSIM=0.933.png}}\caption{VBQ \newline \hspace*{1em} MS-SSIM=$0.933$}
    \end{subfigure}
    \begin{subfigure}[h]{0.19\textwidth}
    {\includegraphics[width=\textwidth]{figs/img_compression/reconstructions/flowers.png-UniformQuantizer-BPP=0.243-PSNR=16.557-MSSSIM=0.723.png}}\caption{Uniform grid \newline \hspace*{1em} MS-SSIM=$0.723$}
    \end{subfigure}
    \begin{subfigure}[h]{0.19\textwidth}
        {\includegraphics[width=\textwidth]{figs/img_compression/reconstructions/flowers_BPP=0.236375.png.tfci.png}}\caption{Ball{\'e} et al. \newline \hspace*{1em} MS-SSIM=$0.958$}
    \end{subfigure}

\caption{Image reconstructions at matching bitrate (0.24 bits per pixel).
VBQ~(c; proposed) outperforms AC with uniform quantization~(d) and JPEG~(b) and is comparable to the approach by \citep{balle2017end}~(e) despite using a model that is not optimized for this specific bitrate. Uniform quantization here used a modified version of the VAE in Figure~\ref{fig:kodak-rd},
using an additional conv layer with smaller dimensions to reduce the bitrate down to 0.24
(this was not possible in the original model even with the largest possible grid spacing).
}
\label{fig:example-reconstructions}
\end{figure*}

While Section~\ref{sec:word-embeddings} demonstrated the proposed VBQ method for model compression, we now apply the same method to data compression using a variational autoencoder (VAE).
We first provide
qualitative insight on small-scale images, and then
quantitative results on full resolution color images.

\paragraph{Model.}
For simplicity, we consider regular VAEs with a standard normal prior
and Gaussian variational posterior.
The generative network parameterizes a factorized categorical or Gaussian likelihood model in experiments in Sec. \ref{sec:toy} or \ref{sec:kodak},
respectively. Network architectures are described below and in more detail in Supplementary Material.

\paragraph{Baselines.}

We consider the following baselines:
\begin{itemize}
\setlength\itemsep{-.1em}
    \item \emph{Uniform quantization:} for a given image~${\mathbf x}$, we quantize each dimension of the posterior mean vector $\boldsymbol{\mu}({\mathbf x})$ to a uniform grid.
    We report the bitrate for encoding the resulting quantized latent representation via standard entropy coding (e.g., arithmetic coding).
    Entropy coding requires prior knowledge of the probabilities of each grid point.
    Here, we use the empirical frequencies of grid points over a subset of the training set;
    \item \emph{$k$-means quantization}: similar to ``uniform quantization'', but with the placement of grid points optimized via $k$-means clustering on a subset of the training data;
    \item Quantization with \emph{generalized Lloyd algorithm}: similar to above, but the grid points are optimized using generalized Lloyd algorithm \citep{Chou1989EntropyconstrainedVQ}, a widely-used state-of-the-art classical quantization method;
    \item \emph{JPEG:} we used the libjpeg implementation packaged with the Python Pillow library, using default configurations (e.g., 4:2:0 subsampling), and we adjust the quality parameter to vary the rate-distortion trade-off;
    \item \emph{Deep learning baseline:} we compare to \citet{balle2017end}, who directly optimized for the rate and distortion, training a separate model for each point on the R-D curve.
    In our large-scale experiment, we adopte their
    model architecture,
    so their performance essentially represents the end-to-end optimized performance upper bound for our method (which uses a single model).
\end{itemize}

\subsubsection{Qualitative Analysis on Toy Datasets}
\label{sec:toy}

We trained a VAE on the MNIST dataset and the Frey Faces dataset, using 5 and 4-dimensional latent spaces, respectively. See Supplemental Material for experimental details.

Figure \ref{fig:mnist-series} shows example image reconstructions from our VBQ algorithm with increasing $\lambda$, and thus decreasing bitrate. The right-most column is the extreme case $\lambda \to \infty$, resulting in the shortest possible bistring encoding $\hat \xi_i = (0.1)_2 = \frac12$ (i.e., $\hat z_i$ being the median of the prior $p(z_i)$) for every dimension $i$.
As the bitrate decreases (as ${\mathcal R}(\boldsymbol{\hat\xi}) \to 0$), our method gradually ``confuses" the original image with a generic image (roughly in the center of the embedding space), while preserving approximately the same level of sharpness.
This is in contrast to JPEG which typically introduces blocky and/or pixel-level artifacts at lower bitrates.

\subsubsection{Full-Resolution Color Images} \label{sec:kodak}
We apply our VBQ method to a VAE trained on color images, and
obtain practical image compression performance rivaling JPEG, while outperforming baselines that ignore posterior uncertainty and directly quantize latent variables.

\paragraph{Model and Dataset.}
The inference and generative networks of our VAE are identical to the analysis and synthesis networks of \citet{balle2017end}, using 3 layers of 256 filters each in a convolutional architecture.
We used a diagonal Gaussian likelihood model, whose mean is computed by the generative net and the variance $\sigma^2$ is fixed as a hyper-parameter, similar to a $\beta$-VAE \citep{higgins2016beta} approach ($\sigma^2$ was tuned to 0.001 to ensure the VAE achieved overall good R-D trade-off; see \citep{alemi2018fixing}).
We trained the model on the same subset of the ImageNet dataset as used in \citep{balle2017end}.
We evaluated performance on the standard Kodak \citep{kodak-cd} dataset,
a separate set of 24 uncompressed color images.
As in the word embedding experiment, we also observed that using an empirical prior for our method improved the bitrate; for this,
we used the flexible density model of \citet{balle2018variational},
fitting a different distribution
per latent channel, on samples of posterior means $\boldsymbol\mu$ (treating spatial dimensions as i.i.d.).

\begin{figure}[t]
\centering
 \centering
 \includegraphics[width=0.5\textwidth]{figs/img_compression/kodak_rd.pdf}
\caption{Aggregate rate-distortion performance on the Kodak dataset (higher is better).
VBQ (blue, proposed) outperforms JPEG for all tested bitrates with a single model.
By contrast, \citep{balle2017end} (black squares) relies on individually optimized models for each bitrate that all have to be included in the decoder.
}\label{fig:kodak-rd}
\end{figure}

\paragraph{Results.}
As common in image compression work, we measure distortion between original and compressed images under two quality metrics (the higher the better): Peak Signal-to-Noise ratio (PSNR) and MS-SSIM \citep{wang2003multiscale}, over all RGB channels.
Figure~\ref{fig:kodak-rd} shows rate-distortion performance.
We averaged both the bits per pixel (BPP) and quality measure over all images in the Kodak dataset for each~$\lambda$
(we got similar results by averaging only over the quality metrics for fixed bitrates via interpolation).

We found that our method generally produced images with higher quality, both in terms of PSNR and perceptual quality, compared to JPEG and uniform quantization. Similar to  \citep{balle2017end}, our method avoids unpleasant artifacts, and introduces blurriness at low bitrate. See Figure \ref{fig:example-reconstructions} for example image reconstructions. For more examples and R-D curves on individual images, see Supplementary Material.

Although our results fall short of the end-to-end optimized rate-distortion performance of \citet{balle2017end}, it is worth emphasizing that our method operates anywhere on the R-D curve with a \emph{single} trained VAE model, unlike \citet{balle2017end}
, which
requires costly optimization and storage of individual models for each point on the R-D curve.
On the other hand, as with any quantization method, the reconstruction quality of VBQ is upper-bounded by that of the full-precision latents; e.g., evaluating on uncompressed latents gives a PSNR upper-bound of about 38.9 in Figure~\ref{fig:kodak-rd}.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\columnwidth]{figs/img_compression/posterior_collapse.pdf}
\end{center}
\caption{Variational posteriors and the encoding cost for the first~6 latent channels of an image-compression VAE trained with high~$\beta$ setting.
``Baseline'' refers to uniform quantization.
The proposed VBQ method wastes fewer bits on channels that exhibit posterior collapse (channels 2, 3, and 5) than the baseline method.
It instead spends more bits on channels without posterior collapse.
}
\label{fig:posterior_collapse}
\end{figure}

\paragraph{Indifference to Posterior Collapse.}
A known issue in deep generative models such as VAEs is
posterior collapse, where the model
ignores some subset of latent variables,
whose
variational posterior distributions collapse to closely match the prior.
Such collapsed dimensions constitute an overhead in conventional neural compression approaches,
which is often dealt with by a pruning step.
One curious consequence of our approach is that it automatically spends close to zero bits encoding the collapsed latent dimensions.

As an illustration, we trained a VAE as used in the color image compression experiment with a high $\beta$ setting to purposefully induce posterior collapse, and examine the average number of bits spent on various latent channels. %
Figure \ref{fig:posterior_collapse} shows the prior $p(z_i)$, aggregated (approximate) posterior $q(z_i) :=\mathbb{E}_{\mathbf x}[q(z_i|{\mathbf x})]$, and histograms of posterior means $\mu_i({\mathbf x})$
for the first six channels of the VAE; all the quantities were averaged over an image batch and across latent spatial dimensions.
We observe that channels 2, 3, and 5 appear to exhibit posterior collapse, as the aggregated posteriors closely match the prior while the
posterior means tightly cluster at zero;
this is also reflected by low average KL-divergence between the variational posterior $q(z_i|{\mathbf x})$ and prior $p(z_i)$, see text inside each panel.
We observe that, for these collapsed channels, our method spends fewer bits on average than uniform quantization (baseline) at the same total bitrate, and more bits
instead on channels 1, 4, and 6, which do not exhibit posterior collapse.
The explanation is that a collapsed posterior has unusually high variance $\sigma_i^2({\mathbf x})$, causing our model to refrain from long code words due to the high penalty $\propto \sigma_i^2({\mathbf x})$ per bitrate ${\mathcal R}(\hat\xi_i)$ in Eq.~\ref{eq:objective-single-dim}.

\section*{Acknowledgements}
We thank Johannes Ball{\'e} for helping us reproduce the results in prior work \citep{balle2017end}. Yibo Yang acknowledges funding from the Hasso Plattner Foundation. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021.  Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA). Furthermore, this work was supported by the National Science Foundation under Grants NSF-1928718, NSF-2003237, and by Qualcomm.

\section{Posterior-Informed Variable-Bitrate Compression}
\label{sec:method}

We now propose an algorithm for quantizing latent variables in trained models.
After describing the problem setup and assumptions (Subsection~\ref{sec:models}), we briefly review
Arithmetic Coding (Subection~\ref{sec:arithmetic-coding}).
Subsection~\ref{sec:bayesian-ac} describes our proposed lossy compression algorithm, which generalizes Arithmetic Coding to the continuous domain.

\subsection{Problem Setup}
\label{sec:models}

\paragraph{Generative Model and Variational Inference.}
We consider a wide class of generative probabilistic models with data~${\mathbf x}$ and unknown (or ``latent'') variables ${\mathbf z}\in\mathbb R^K$ from some continuous latent space with dimension~$K$.
The generative model is defined by a joint probability distribution,
\begin{align}\label{eq:joint-distribution}
    p({\mathbf x},{\mathbf z}) = p({\mathbf z}) \, p({\mathbf x}|{\mathbf z})
\end{align}
with a prior $p({\mathbf z})$ and a likelihood $p({\mathbf x}|{\mathbf z})$. Although our presentation focuses on unsupervised representation learning, our framework also captures the supervised setup.%
\footnote{For supervised learning with labels~$y$, we would consider a conditional generative model $p(y,{\mathbf z}|{\mathbf x}) =  p(y|{\mathbf z},{\mathbf x})\, p({\mathbf z})$ with conditional likelihood $p(y|{\mathbf z},{\mathbf x})$, where ${\mathbf z}$ are the model parameters, treated as a Bayesian latent variable with associated prior $p({\mathbf z})$.}

Our proposed compression method uses~${\mathbf z}$ as a proxy to describe the data~${\mathbf x}$.
This requires ``solving'' Eq.~\ref{eq:joint-distribution} for~${\mathbf z}$ given~${\mathbf x}$, i.e., inferring the posterior
$p({\mathbf z}|{\mathbf x}) = p({\mathbf x},{\mathbf z}) / \int\! p({\mathbf x},{\mathbf z})\, d{\mathbf z}$.
Since exact Bayesian inference is often intractable,  %
we resort to Variational Inference (VI)~\citep{jordan1999introduction,blei2017variational, zhang2019advances}, which approximates the posterior by a so-called variational distribution $q_\phi({\mathbf z}|{\mathbf x})$ by minimizing the Kullback-Leibler divergence $D_\text{KL}(q_\phi({\mathbf z}|{\mathbf x}) \,||\, p({\mathbf z}|{\mathbf x}))$ over a set of variational parameters~$\phi$.

\paragraph{Factorization Assumptions.}\label{par:factorization-assumptions}
We assume that both the prior $p({\mathbf z})$ and the variational distribution $q_\phi({\mathbf z}|{\mathbf x})$ are fully factorized (mean-field assumption).
For concreteness, our examples use a Gaussian variational distribution.
Thus,
\begin{align}%
    p({\mathbf z}) &= \textstyle{\prod}_{i=1}^K p(z_i); \label{eq:factorized-prior}
    \qquad\text{and} \\
    q_\phi({\mathbf z}|{\mathbf x}) &= \textstyle{\prod}_{i=1}^K \mathcal N(z_i; \mu_i({\mathbf x}), \sigma^2_i({\mathbf x})), \label{eq:factorized-q}
\end{align}
where $p(z_i)$ is a prior for the $i$\textsuperscript{th} component of~${\mathbf z}$, and the means~$\mu_i$ and standard deviations~$\sigma_i$ together comprise the variational parameters~$\phi$ over which VI optimizes.%
\footnote{These parameters are often amortized by a neural network (in which case $\mu_i$ and $\sigma_i$ depend on~${\mathbf x}$), but don't have to (in which case $\mu_i$ and $\sigma_i$ do not depend on ${\mathbf x}$ and are directly optimized).}

Prominently, the model class defined by Eqs.~\ref{eq:joint-distribution}-\ref{eq:factorized-q} includes variational autoencoders (VAEs)~\citep{KW2014} for data compression, but we stress that the class is much wider, capturing also Bayesian neural nets \citep{mackay2003information},
probabilistic word embeddings \citep{barkan2017bayesian,bamler2017dynamic}, matrix factorization \citep{mnih2008probabilistic}, and topic models \citep{blei2003latent}.

\paragraph{Protocol Overview.}
We consider two parties in communication, a sender and a receiver.
In the case of \emph{data} compression, both parties have access to the model, but only the sender has access to the data point~${\mathbf x}$, which it uses to fit a variational distribution $q_\phi({\mathbf z}|{\mathbf x})$. %
It then uses the algorithm proposed below to select a latent variable vector~${\mathbf{\hat z}}$ that has high probability under $q_\phi$, and that can be encoded into a compressed bitstring, which gets transmitted to the receiver.
The receiver  losslessly decodes the compressed bitstring back into~${\mathbf{\hat z}}$ and uses the likelihood $p({\mathbf x}|{\mathbf{\hat z}})$ to generate a reconstructed data point ${\mathbf{\hat x}}$, typically setting ${\mathbf{\hat x}} = \arg \max_{\mathbf x} p({\mathbf x}|{\mathbf{\hat z}})$.
 In the case of \emph{model} compression, the sender infers a distribution $q_\phi({\mathbf z}|{\mathbf x})$ over model parameters ${\mathbf z}$ given training data ${\mathbf x}$, and uses our algorithm to select a suitable vector ${\mathbf{\hat z}}$ of quantized model parameters. The receiver receives~${\mathbf{\hat z}}$ and uses it to reconstruct the model.
\label{compression-protocal}

The rest of this section describes how the proposed algorithm selects~${\mathbf{\hat z}}$ and encodes it into a compressed bitstring.

\subsection{Background: Arithmetic Coding}
\label{sec:arithmetic-coding}

Our quantization algorithm, introduced in Section~\ref{sec:bayesian-ac} below, is inspired by a lossless compression algorithm, \emph{arithmetic coding} (AC) \citep{witten1987arithmetic, mackay2003information}, which we generalize from discrete data to the continuous space of latent variables ${\mathbf z}\in\mathbb R^K$.
To get there, we first review the main idea of AC that our proposed algorithm borrows.

AC is an instance of so-called entropy coding.
It uniquely maps messages~$m\in \mathcal M$ from a discrete set~$\mathcal M$ to a compressed bitstring of some length~${\mathcal R}_m$ (the ``bitrate'').
Entropy coding exploits prior knowledge of the distribution $p(m)$ of messages to map probable messages to short bitstrings while spending more bits on improbable messages.
This way, entropy coding algorithms aim to minimize the expected rate $\mathbb E_{p(m)}[\mathcal R_m]$.
For lossless compression, the expected rate has a fundamental lower bound, the entroy $H=\mathbb E_{p(m)}[h(m)]$, where $h(m)=-\log_2 p(m)$ is the Shannon information content of~$m$.
AC provides near optimal lossless compression as it maps each message~$m\in\mathcal M$ to a bitstring of length ${\mathcal R}_m=\lceil h(m)\rceil$, where $\lceil\cdot\rceil$ denotes the ceiling function.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\columnwidth]{figs/method/ac-encoding.pdf}
\end{center}
\caption{Comparison of Arithmetic Coding (AC, left) and VBQ (right, proposed).
Both methods use a prior CDF (orange) to map nonuniformly distributed data to a number $\xi\sim \mathcal U(0,1)$, and both require an uncertainty region for truncation.}
\label{fig:ac}
\end{figure}

AC is usually discussed in the context of streaming compression where $m$ is a sequence of symbols from a finite alphabet, as AC improves on this task over the more widely known Huffman coding~\citep{huffman1952method}.
In our work, we focus on a different aspect of AC: its use of a cumulative probability distribution function to map a nonuniformly distributed random variable $m \sim p(m)$ to a number~$\xi$ that is nearly uniformly distributed over the interval $[0,1)$.

Figure~\ref{fig:ac} (left) illustrates AC for a binomial-distributed message $m\in\{0,\ldots,10\}$ (the number of `heads' in a sequence of ten coin flips).
The solid and dashed orange lines show the left and right sided cumulative distribution function,%
\footnote{If $m$ is a sequence of symbols, $F_<$ and $F_\leq$ are defined by lexicographical order and can be constructed in a streaming manner.}
$F_<(m):=\sum_{m'<m} p(m')$ and $F_\leq(m):=\sum_{m'\leq m} p(m')$,
respectively.
They define a partitioning of the interval $[0,1)$ (vertical axis in Figure~\ref{fig:ac} (left)) into pairwise disjoint subintervals $\mathcal I_m := \big[F_<(m), F_\leq(m)\big) $ (orange squares).
Since the intervals $\mathcal I_m$ are disjoint for all $m\in\mathcal M$, any number $\xi\in\mathcal I_m$ uniquely identifies a given message~$m$.
AC picks such a number~$\hat\xi \in \mathcal I_m$ and encodes it into a string of bits~$b_\kappa$, $\kappa \in \{0, \ldots, {\mathcal R}_m\}$ by writing it in binary representation,
\begin{align}\label{eq:xi-m}
    \hat\xi = {\left(0.b_1b_2\ldots b_{{\mathcal R}_{m}}\right)}_2
    \quad\text{with bits $b_\kappa\in\{0,1\}\; \forall \kappa$.}
\end{align}

Since any $\xi\in\mathcal I_m$ may be used to identify the message~$m$, we can interpret the interval~$\mathcal I_m$ as an \emph{uncertainty region} in $\xi$-space.
AC picks the number $\hat\xi\in\mathcal I_m$ with the shortest binary representation.
This requires at most $\lceil h(m)\rceil$ bits because the numbers~$\xi$ that can be represented by Eq.~\ref{eq:xi-m} with ${\mathcal R}_m=\lceil h(m)\rceil$ form a uniform grid with spacing $2^{-{\mathcal R}_{m}}=2^{-\lceil h(m)\rceil}$, which is at most as wide as the size of the interval, $|\mathcal I_{m}|=p(m)=2^{-h(m)}$.
The red arrows in Figure~\ref{fig:ac} (left) illustrate how AC would encode the message $m=7$ in the toy example into the bitstring ``$111$''.
Decoding works in the opposite direction and maps $\hat\xi$ back to~$m$.

In the next section, we generalize AC to the continuous domain.
As we will show, the concept of an ``uncertainty region'' in $\xi$-space becomes again crucial.

\subsection{Variational Bayesian Quantization}
\label{sec:bayesian-ac}

We now present our proposed algorithm, Variational Bayesian Quantization (VBQ), a novel quantization method for lossy compression that is inspired by AC but that operates on \emph{continuous} latent variables ${\mathbf z}\in\mathbb R^K$.
Similar to AC, VBQ exploits knowledge of a prior probability distribution $p({\mathbf z})$ in combination with a (soft) uncertainty region to encode probable values of~${\mathbf z}$ into short bitstrings.

\paragraph{From Intervals to Distributions.}
The main ideas that VBQ borrows from AC are as follows: (1)~the use of a cumulative distribution function to map a non-uniformly distributed random variable to a uniformly distributed random variable $\xi$ over the interval $(0,1)$, and (2)~the use of an ``uncertainty region" to select a number from this interval to encode the message with as few bits as possible. While AC uses an interval~$\mathcal I_m$ with hard boundaries, VBQ softens this uncertainty region
by considering posterior uncertainty.

We consider a single continuous latent variable $z_i\in\mathbb R$ with arbitrary prior $p(z_i)$.
The cumulative (CDF) of the prior, %
\begin{align} %
    F(z_i) := \int_{-\infty}^{z_i} p(z_i')\, d z_i'\, ,
\end{align}
is shown in orange in Figure~\ref{fig:ac} (right).
It maps $z_i \sim p(z_i)$ to $\xi_i \sim \mathcal U(0,1)$.
In contrast to the discrete case discussed in Section~\ref{sec:arithmetic-coding}, where the prior CDF maps each message~$m$ to an entire interval $\mathcal I_m$, note that the CDF of a continuous random variable maps real numbers to real numbers.

Since $\xi_i\sim \mathcal U(0,1)$ is almost surely an irrational number, its binary representation is infinitely long, and thus has to be truncated.
We find an optimal truncation by generalizing the idea of the uncertainty region~$\mathcal I_m$ to the continuous space:
we consider the posterior uncertainty in $z_i$-space and map it to $\xi_i$-space.
Approximating the posterior $p(z_i|{\mathbf x})$ by the variational distribution $q(z_i|{\mathbf x}) := \mathcal N(z_i;\mu_i({\mathbf x}),\sigma_i^2({\mathbf x}))$, see Eq.~\ref{eq:factorized-q}, we thus consider the function
\begin{align}\label{eq:def-g}
    g(\xi_i) := q(F^{-1}(\xi_i) \,|\, {\mathbf x}).
\end{align}
Here, $F^{-1}$ is the inverse CDF (the quantile function), which maps $\xi_i$ back to~$z_i$.
Note that $g$ is not a normalized probability distribution, as Eq.~\ref{eq:def-g} deliberately does not include the Jacobian $\nabla_{\!\xi_i}F^{-1}(\xi_i)$ because the final objective will be to maximize $q_\phi({\mathbf z}|{\mathbf x})$ at a single point (see Eq.~\ref{eq:log-q-log-g} below).

\paragraph{Intuition.}
The solid and dashed purple curves in figure Figure~$\ref{fig:ac}$ (right) plot $q(z_i|{\mathbf x})$ and $g(\xi_i)$ on the horizontal and vertical axis, respectively.
The red arrows illustrate how a finite uncertainty region $\mu_i({\mathbf x}) \pm \sigma_i({\mathbf x})$ in $z_i$-space is mapped to a finite width of~$g$ in $\xi_i$-space.
VBQ finds a quantile~$\hat\xi_i$ that has high value under $g$ while at the same time having a short binary representation.
The two purple arrowheads on the vertical axis point to two viable candidates, $\hat \xi_i=\frac78$ and $\hat \xi_i=\frac34$, that both lie within the uncertainty region.
The choice between these two points poses a rate-distortion trade-off:
while $\frac78 \equiv {(0.111)}_2$ has higher value under~$g$ (i.e., it identifies a point $\hat z_i= F^{-1}(\frac78)$ with higher approximate posterior probability $q(\hat z_i|{\mathbf x})$), the alternative $\hat\xi_i=\frac34 \equiv {(0.11)}_2$ can be encoded in fewer bits.

\paragraph{Optimizing the Rate-Distortion Trade-Off.}
Rather than considering a hard uncertainty region, VBQ simply tries to find a point $\boldsymbol\xi\equiv(\xi_i)_{i=1}^K$ that identifies latent variables ${\mathbf z}\equiv (z_i)_{i=1}^K$ with high probability under the variational distribution $q_\phi({\mathbf z}|{\mathbf x})$ while being expressible in few bits.
We thus express $\log q_\phi({\mathbf z}|{\mathbf x})$ in terms of the coordinates $\xi_i = F(z_i)$ using Eq.~\ref{eq:factorized-q},
\begin{align}\label{eq:log-q-log-g}
    \log q_\phi({\mathbf z}|{\mathbf x})
    &=-\sum_{i=1}^K \frac{\big( F^{-1}( \xi_i) - \mu_i({\mathbf x})\big)^2}{2\,\sigma_i^2({\mathbf x})} + \text{cnst.}
\end{align}

For each dimension~$i$, we restrict the quantile~$\xi_i \in (0, 1)$ to the set of code points $\hat\xi_i$ that can be represented in binary via Eq.~\ref{eq:xi-m} with a finite but arbitrary bitlength ${\mathcal R}(\hat\xi_i)$.
We define the total bitlength ${\mathcal R}(\boldsymbol{\hat\xi}):=\sum_{i=1}^K{\mathcal R}(\hat\xi_i)$, i.e., the length of the concatenation of all codes $\hat\xi_i$, $i\in\{1,\ldots,K\}$ neglecting, for now, an overhead for delimiters (see below).
Using a rate penalty parameter $\lambda>0$ that is shared across all dimensions~$i$, we minimize the rate-distortion objective
\begin{align}\label{eq:rd-objective}
      \mathcal L_\lambda(\boldsymbol{\hat\xi}|{\mathbf x})
      &= -\log q_\phi({\mathbf{\hat z}}|{\mathbf x}) + \lambda {\mathcal R}(\boldsymbol{\hat\xi}) \\
      &= \sum_{i=1}^K \left[ \frac{\big(F^{-1}(\hat \xi_i) - \mu_i({\mathbf x})\big)^2}{2\,\sigma_i^2({\mathbf x})} + \lambda \mathcal R(\hat \xi_i) \right] + \text{cnst}. \nonumber
\end{align}
The optimization thus decouples across all latent dimensions~$i$, and can be solved efficiently and in parallel by minimizing the $K$ independent objective functions
\begin{align}\label{eq:objective-single-dim}
    \ell_\lambda(\hat\xi_i|{\mathbf x})=  \big(F^{-1}(\hat \xi_i) - \mu_i({\mathbf x})\big)^2 + 2\lambda\,\sigma_i^2({\mathbf x}) \, \mathcal R(\hat \xi_i).
\end{align}

\begin{algorithm}[tb]
   \caption{Rate-Distortion Optimization for Dimension $i$}
   \label{alg:scalar-rd-optimization}
\begin{algorithmic}
    \STATE \begin{tabular}{@{}l@{~}l@{}}
    {\bfseries Input:} & Prior CDF $F(z_i)$,
        rate penalty $\lambda>0$, \\
     & variational mode  $\mu_i({\mathbf x})$ and variance $\sigma^2_i({\mathbf x})$. \\
    {\bfseries Output:} & Optimal code point $\hat\xi_i^*\equiv (0.b_1b_2\ldots b_{{\mathcal R}(\hat\xi_i^*)})_2$.
    \end{tabular}
    \STATE Evaluate $\xi_i^\dagger \gets F(\mu_i({\mathbf x}))$.
    \STATE Initialize $r\gets0,\quad \hat\xi_i^* \gets \text{null},\quad
    \ell^* \leftarrow \infty$. \vspace{1pt}
    \REPEAT
        \STATE Update $r\gets r+1$
        \STATE Set $\hat\xi_i^{r,\text{left}} \leftarrow 2^{-r} \lfloor 2^{r} \xi_i^\dagger\rfloor,\quad
        \hat\xi_i^{r,\text{right}} \gets 2^{-r} \lceil 2^{r} \xi_i^\dagger\rceil$.
        \IF{$\hat\xi_i^{r,\text{left}}\neq0$ \textbf{and} $\ell_{\lambda}(\hat\xi_i^{r,\text{left}} \,|\,{\mathbf x}) < \ell^*$  }
          \STATE Update $\hat\xi_i^* \gets \hat\xi_i^{r,\text{left}},\quad
          \ell^* \leftarrow \ell_{\lambda}(\hat\xi_i^{r,\text{left}} \,|\,{\mathbf x})$.
        \ENDIF
        \IF{$\hat\xi_i^{r,\text{right}}\neq1$ \textbf{and} $\ell_{\lambda}(\hat\xi_i^{r,\text{right}} \,|\,{\mathbf x}) < \ell^*$  }
          \STATE Update $
          \hat\xi_i^* \gets \hat\xi_i^{r,\text{right}},\quad
          \ell^* \leftarrow \ell_{\lambda}(\hat\xi_i^{r,\text{right}} \,|\,{\mathbf x})$.
        \ENDIF

    \UNTIL{$\log g(\xi_i^\dagger) -\log g(\hat\xi_i^*) < \lambda (r+1-{\mathcal R}(\hat\xi_i^*))$.}
\end{algorithmic}
\end{algorithm}

Although the bitlength $R(\hat \xi_i)$ is discontinuous (it counts the number of binary digits, see Eq.~\ref{eq:xi-m}), $\ell_\lambda(\hat\xi_i|{\mathbf x})$ can be efficiently minimized over~$\hat\xi_i$ using Algorithm \ref{alg:scalar-rd-optimization}.
The algorithm iterates over all rates $r\in\{1,2,\ldots\}$ and searches for the code point~$\hat\xi_i^*$ that minimizes $\ell(\hat\xi_i|{\mathbf x})$.
For each~$r$, the algorithm only needs to consider the two code points $\hat\xi_i^{r,\text{left}} \leq \xi_i^\dagger$ and $\hat\xi_i^{r,\text{right}} \geq \xi_i^\dagger$ with rate at most $r$ that enclose the optimum $\xi_i^\dagger:= F(\mu_i({\mathbf x}))$ and are closest to it; these two code points can be easily computed in constant time.
The iteration terminates as soon as the maximally possible remaining increase in $\log q(z_i|{\mathbf x})=\log g(\xi_i)$ is smaller than the minimum penalty for an increasing bitlength (in practice, the iteration rarely exceeds $r\approx 8$).

\paragraph{Encoding.}
After finding the optimal code points $(\hat\xi_i^*)_{i=1}^K$, they have to be encoded into a single bitstring.
Simply concatenating the binary representations (Eq.~\ref{eq:xi-m}) of all $\hat\xi_i^*$ would be ambiguous due to their variable lengths ${\mathcal R}(\hat\xi_i^*)$
(see detailed discussion in the Supplementary Material).
Instead, we treat the code points as symbols from a discrete vocabulary and encode them via lossless entropy coding, e.g., Arithmetic Coding.
The entropy coder requires a probabilistic model over all code points; here we simply use their empirical distribution.
When using our method for model compression, this empirical distribution has to be transmitted to the receiver as additional header information that counts towards the total bitrate.
For data compression, by contrast, we obtain the empirical distribution of code points on training data %
and include it in the decoder.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\columnwidth]{figs/method/anisotropy.pdf}
\end{center}
\caption{Effect of an anisotropic posterior distribution on quantization.
Left: linear regression model with optimal fit (green) and fits of models with quantized parameters (orange, purple).
Right: posterior distribution and quantized model parameters following two different quantization schemes.
Although both quantized models are equally far away from the optimal solution (green dot), VBQ (orange) fits the data better because it takes the anisotropy of the posterior into account.
}
\label{fig:anisotropy}
\end{figure}

\paragraph{Discussion.}\label{par:method-discussion}
The proposed %
algorithm adjusts the accuracy for each latent variable~$z_i$ based on two factors:
(i)~a \textit{global} rate setting $\lambda$ that is shared across all dimensions~$i$; and
(ii)~a per-dimension posterior uncertainty estimate $\sigma_i({\mathbf x})$.
Point~(i) allows tuning the rate-distortion trade-off whereas (ii)~takes the anisotropy of the latent space into account.

Figure~\ref{fig:anisotropy} illustrates the effect of anisotropy in latent space.
The right panel plots the posterior of a toy Bayesian linear regression model $y=ax+b$ (see left panel) with only two latent variables ${\mathbf z}\equiv(a,b)$.
Due to the elongated shape of the posterior, VBQ uses a higher accuracy for~$a$ than for~$b$.
As a result, the algorithm finds a quantization~${\mathbf{\hat z}}$ (orange dot in right panel) that is closer to the optimal (MAP) solution (green dot) along the $a$-axis than along the $b$-axis.

The purple dot in Figure~\ref{fig:anisotropy} (right) compares to a more common quantization method, which simply
rounds the MAP solution to the nearest point (which is then entropy coded) from a fixed grid with spacing $\delta>0$.
We tuned $\delta$ so that the resulting quantized model parameters (purple dot) have the same distance to the optimum as our proposed solution (orange dot).
Despite the equal distance to the optimum, VBQ finds model parameters with higher posterior probability. The resulting model fits the data better (left panel).

This concludes the description of the proposed Variational Bayesian Quantization algorithm.
In the next section, we analyze the algorithm's behaviour experimentally and demonstrate its performance for variable-bitrate compression on both word embeddings and images.

\section{Introduction}

Latent-variable models have become a mainstay of modern machine learning.
Scalable approximate Bayesian inference methods, in particular Black Box Variational Inference \citep{ranganath2014black, rezende2014stochastic}, have spurred the development of increasingly large and expressive probabilistic models, including deep generative probabilistic models such as variational autoencoders \citep{kingma2013auto} and Bayesian neural networks \citep{mackay1992practical, blundell2015weight}.
~One natural application of deep latent variable modeling is data compression, and recent work has focused on end-to-end procedures that optimize a model for a particular compression objective.
Here, we study a related but different problem: given a \emph{trained} model, what is the best way to encode the information contained in its continuous latent variables?

As we show, our proposed solution provides a new ``plug-and-play'' approach to %
lossy compression of both \emph{data instances} (represented by local latent variables, e.g., in a VAE) as well as \emph{model parameters} (represented by global latent variables that serve as parameters of a Bayesian statistical model).
Our approach separates the compression task from model design and training, thus implementing variable-bitrate compression as an independent post-processing step in a wide class of existing latent variable models.

At the heart of our proposed method lies a novel quantization scheme that optimizes a rate-distortion trade-off by exploiting posterior uncertainty estimates.
Quantization is central to lossy compression, as continuous-valued data like natural images, videos, and distributed representations ultimately need to be discretized to a finite number of bits for digital storage or transmission.
Lossy compression algorithms therefore typically find a discrete approximation of some semantic representation of the data, which is then encoded with a lossless compression method.

In classical lossy compression methods such as JPEG or MP3, the semantic representation is carefully designed to support compression at variable bitrates.
By contrast, state-of-the-art deep learning based approaches to lossy data compression \citep{balle2017end,balle2018variational,rippel2017realtime,mentzer2018conditional,lombardo2019deep} are trained to minimize a distortion metric at a fixed bitrate.
To support variable-bitrate compression in this approach, one has to train several models for different bitrates.
While training several models may be viable in many cases, a bigger issue is the increase in decoder size as the decoder has to store the parameters of not one but several deep neural networks for each bitrate setting.
In applications like video streaming under fluctuating connectivity, the decoder further has to load a new deep learning model into memory every time a change in bandwidth requires adjusting the bitrate.

By contrast, we propose a
a quantization method for latent variable models
that decouples training from compression, and that enables variable-bitrate compression with a single model.
We generalize a classical entropy coding algorithm, Arithmetic Coding \citep{witten1987arithmetic, mackay2003information}, from the discrete to continuous domain.
Our proposed algorithm, Variational Bayesian Quantization, exploits posterior uncertainty estimates to automatically reduce the quantization accuracy of latent variables for which the model is uncertain anyway.
This strategy is analogous to the way humans communicate quantitative information.
For example, Wikipedia lists the population of Rome in 2017 with the specific number $2{,}879{,}728$.
By contrast, its population in the year $500$~AD is estimated by the round number $100{,}000$ because
the high uncertainty would make a more precise number meaningless.
Our ablation studies show that this posterior-informed quantization scheme is crucial to obtaining competitive performance.

In detail, our contributions are as follows:
\begin{itemize}
    \item \textit{A new discretization scheme.}
    We present a novel approach to discretizing latent variables in a variational inference framework. Our approach generalizes arithmetic coding from discrete to continuous distributions and takes posterior uncertainty into account.
    \item \textit{Single-model compression at variable bitrates.}
    The decoupling of modeling and compression allows us to adjust the trade-off between bitrate and distortion in post-processing.
    This is in contrast to existing approaches to both data and model compression, which often require specially trained models for each bitrate.
    \item \textit{Automatic self-pruning.}
    Deep latent variable models often exhibit posterior collapse, i.e., the variational posterior collapses to the model prior.
    In our approach, latent dimensions with collapsed posteriors require close to zero bits, thus don't require manual pruning.
    \item \textit{Competitive experimental performance.}
    We show that our method outperforms JPEG over a wide range of bitrates using only a single model. We also show that we can
    successfully compress word embeddings with minimal loss, as evaluated on semantic reasoning task.
\end{itemize}

The paper is structured as follows:
Section~\ref{sec:related} reviews related work in neural compression; Section~\ref{sec:method} proposes our Variational Bayesian Quantization algorithm. We give empirical results %
in Section~\ref{sec:experiments}, and conclude in
Section~\ref{sec:conclusions}.
~ Section~\ref{sec:theory} provides additional theoretical insight about our method.

\section{Delimitation Overhead}
\label{app:practical-details}

We elaborate on the ``encoding'' paragraph of Section~3.3 of the main paper.
After finding a quantized code point~$\hat\xi_i$ for each dimension $i\in\{1,\ldots,K\}$ of the latent space, these code points have to be losslessly encoded into a single bitstring for transmission or storage.
We experimented with two encoding schemes, described in Subsections~\ref{app:alternative-encoding} and~\ref{app:used-encoding} below.
Subsection~\ref{app:rate-vs-inf-content} provides further analysis.

\subsection{Encoding via Concatenation}
\label{app:alternative-encoding}

We first describe an encoding scheme that we did not end up using, but that makes it easier to understand the objective function of VBQ (Eq.~8 of the main text).
This encoding scheme concatenates the binary representations of $\hat\xi_i$ (Eq.~4 of the main text) for all $i\in\{1,\ldots,K\}$ in to a single bitstring.
As each dimension~$i$ contributes ${\mathcal R}(\hat\xi_i)$ bits to the concatenated bitstring, this encoding scheme justifies the rate penalty term ``$\lambda{\mathcal R}(\hat\xi_i)$'' in Eq.~4 of the main text.

One also has to transmit the rates ${\mathcal R}(\hat\xi_i)$ (in compressed form using traditional entropy coding) so that the decoder can split the concatenated bitstring at the correct positions.
While this incurs some overhead, the variable-bitlength representation of $\hat\xi_i$ also saves one bit per dimension~$i$ because the last bit in the binary representation of each~$\hat\xi_i$ does not need to be transmitted as it is always equal to one (otherwise, the optimization algorithm in VBQ would favor an equivalent shorter binary representation of~$\hat\xi_i$).

\subsection{Encoding via Standard Entropy Coding}
\label{app:used-encoding}

The actual encoding scheme we ended up using does not deal with the binary representation of each~$\hat\xi_i$ explicitly.
Instead, we treat each~$\hat\xi_i$ as a discrete symbol and directly encode the sequence $(\hat\xi_i)_{i=1}^K$ of symbols via entropy coding (e.g., standard arithmetic coding).
The entropy coder needs a model of the probability $p(\hat\xi_i)$ of each symbol.
For model compression, we use the empirical frequencies, which we transmit as extra header information that counts towards the total bitrate.
For data compression, we estimate the frequencies on training data and include them in the decoder.

Transmitting the empirical frequencies lead to a negligible overhead in the word embeddings experiment.
Only a few hundred code points (depending on~$\lambda$) had nonzero frequencies, so that the compressed file size was dominated by the encoding of $K = Vd = 10^7$ quantized latent variables.

\subsection{Justification of the Rate Penalty Term $\lambda{\mathcal R}(\hat\xi_i)$}
\label{app:rate-vs-inf-content}

All experimental results are reported with the encoding scheme of Section~\ref{app:used-encoding} as it lead to slightly lower bitrates in practice.
A peculiarity of this encoding scheme is that it ignores the length ${\mathcal R}(\hat\xi_i)$ of the binary representation of each~$\hat\xi_i$.
For a sequence of symbols $\boldsymbol{\hat\xi}\equiv (\hat\xi_i)_{i=1}^K$ with an i.i.d.~entropy model $p(\hat\xi_i)$, an optimal entropy coder (such as arithmetic coding) achieves the total bitrate ${\mathcal R}(\boldsymbol{\hat\xi}) = \lceil h(\boldsymbol{\hat\xi}) \rceil$ with the information content
\begin{align}\label{app-eq:total-inf-content}
    h(\boldsymbol{\hat\xi})
    &= \sum_{i=1}^K h(\hat\xi_i)
    = -\sum_{i=1}^K \log_2 p(\hat\xi_i).
\end{align}

In particular, Eq.~\ref{app-eq:total-inf-content} does not depend on ${\mathcal R}(\hat\xi_i)$.
This poses the question whether the rate penalty term ``$\lambda {\mathcal R}(\hat\xi_i)$'' in the VBQ objective (Eq.~8 of the main text) is justified.
Ideally, the algorithm would minimize $\lambda h(\hat\xi_i)=-\lambda \log_2 p(\hat\xi_i)$ instead, but this quantity is unknown until the quantizations~$(\hat\xi_i)_{i=1}^K$ and therefore the empirical frequencies $p(\hat\xi_i)$ are obtained.
Our experiments suggest that ${\mathcal R}(\hat\xi_i)$ is a useful proxy for the eventual value of $h(\hat\xi_i)$.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\columnwidth]{figs/supplement/rate-scatter.pdf}
\end{center}
\caption{Relation between rate estimate ${\mathcal R}(\hat\xi_i)$ and the actual contribution $h(\hat\xi_i)$ of code point~$\hat\xi_i$ to the total bitrate under entropy coding.
The approximate affine linear relationship justifies minimizing ${\mathcal R}(\hat\xi_i)$ as a proxy for $h(\hat\xi_i)$ in VBQ.}
\label{app-fig:rate-scatter}
\end{figure}

Figure~\ref{app-fig:rate-scatter} plots the rate estimate ${\mathcal R}(\hat\xi_i)$, i.e., the integer number of bits in the binary representation of~$\hat\xi_i$ ($x$-axis) against the actual contribution $h(\hat\xi_i)=-\log_2 p(\hat\xi_i)$ to the total bitrate according to Eq.~\ref{app-eq:total-inf-content} ($y$-axis).
The figure shows experimental data for compressed word embeddings at $1.32$~bits per latent dimension.
We make the following observations:
\begin{itemize}
    \item
        For most code points~$\hat\xi_i$, the dependency between $h(\hat\xi_i)$ and ${\mathcal R}(\hat\xi_i)$ can be approximated by an affine linear function, thus justifying the use of ${\mathcal R}(\hat\xi_i)$ in the optimization of Bayesian~AC.
    \item
        The slope of the approximate linear dependency is larger than one.
        This may be understood by the penalty term $\lambda {\mathcal R}(\hat\xi_i)$ in the objective function (Eq.~8 of the main text), which causes the method to avoid code points $\hat\xi_i$ with large rate estimates ${\mathcal R}(\hat\xi_i)$, thus reducing their empirical frequencies $p(\hat\xi_i)$ and increasing their information content $h(\hat\xi_i)=-\log_2 p(\hat\xi_i)$.
        This observation does not invalidate the use of ${\mathcal R}(\hat\xi_i)$ as an estimate for $h(\hat\xi_i)$ since the different slope can be absorbed in a rescaling of the parameter~$\lambda$ %
    \item
        For rates ${\mathcal R}(\hat\xi_i)\geq 4$, there are two code points for each rate with considerably lower information content.
        These code points correspond to the two extremes for each rate, i.e., $\hat\xi_i$~closest to zero or one, respectively.
        The observation that the two extremes have lower information content (i.e., higher empirical frequencies) can be explained by the fact that the empirical prior distribution whose CDF we use to map latent variables~$z_i$ to quantiles~$\xi_i$ does not fully capture the true distribution of variational means.
        Indeed, experiments with a more long tailed empirical prior distribution lead to marginally better performance, but the simplicity of a Gaussian empirical prior seemed more valuable to us.
\end{itemize}

\section{More Experimental Details}
\label{app:exp-details}

\subsection{Word Embeddings}

The word embeddings experiment involved only minimal hyperparameter tuning, and we only optimized for performance of the uncompressed model since the goal of the experiment was to test the proposed compression method on a model that was not tuned for compression.
We trained for $10^5$ iterations with minibatches of $10^4$ randomly drawn words and contexts due to hardware constraints.
We tried learning rates $0.1$ and $1$ and chose $0.1$.

\subsection{Experiments on Images}

As mentioned in the main text, we used regular VAEs in the image experiments with standard normal prior $p({\mathbf z})$ and factorized normal posterior $q({\mathbf z}|{\mathbf x})$ with diagonal covariance.

\subsection{MNIST}
The VAE's inference network has two convolutional layers followed by a fully connected layer.
The two conv layers use 32 and 64 filters respectively, with kernel size 3, stride size 2, and ReLU activation. The fully connected layer has output dimension 10 so that $\boldsymbol\mu$ and $\boldsymbol\sigma^2$ of $q({\mathbf z}|{\mathbf x})$ each has dimension 5.

The generative network architecture mirrors the inference network but in reverse, starting with a dense layer mapping 5 dimensional latent variables to 1568 dimensional, treated as 32-channel 7x7 activations,  and followed by two deconvolutional layers of 64 and 32 filters (with identical padding and stride as the convolutional layers). The output is deconvolved with a single 3x3 filter with sigmoid activation function.
For each pixel, the (scalar) output of the last layer parameterizes the likelihood of the pixel being white.

We trained the network on binarized MNIST images for 100 epochs, using the Adam optimizer with learning rate $10^{-4}$.

\subsection{Frey Faces}
On the Frey Faces dataset, we observe poor reconstruction quality by training on binarized images with a factorized Bernoulli likelihood model; instead, we treat each pixel as an observation from a factorized categorical likelihood model with 256 possible outcomes.

The VAE's inference network has two layers. The first layer flattens the input image, converts each pixel value in $\{0, 1, ..., 255\}$ into a one-hot vector $\in \mathbb{R}^{256}$, and uses it to index a 128-dimensional dense vector. The second layer flattens the result of the first layer as its input (which has dimensionality equal to 128 $\times$  \emph{number of pixels}), and fully connects its input to 8 hidden units. The final output is split to obtain 4-dimensional $\boldsymbol \mu$ and $\log \boldsymbol \sigma^2$ of $q({\mathbf z}|{\mathbf x})$.

The generative network has two fully connected layers. The first layer uses 4 hidden units and ReLU activation; the second layer uses   256 $\times$ \emph{number of pixels} hidden units, and takes a 256-way softmax to compute the categorical probability of of each pixel value taking value in $\{0, 1, ..., 255\}$.

We obtained the Frey Faces images from \url{https://cs.nyu.edu/~roweis/data.html}. We trained on a random subset of 1800 images for 800 epochs, using the Adam optimizer with learning rate $10^{-4}$.

On both MNIST and Frey Faces, we vary the rate-distortion trade-off parameter $\lambda$ of Variational Bayesian Quantization between $10^{-5}$ and $10^4$.

\subsection{ Color Image Compression}

As mentioned in the main text, the VAE here uses a fully convolutional architecture with 3 layers of 256 filters each, the same as in (Ball{\'e} et al., 2017); see the latter for detailed descriptions.
We tuned the variance $\sigma^2$ of the likelihood model on a logarithmic grid from $10^{-4}$ to $0.1$ and set it to $0.001$. The VAE was trained on the same dataset as in (Ball{\'e} et al., 2017) for 2 million steps, using Adam with learning rate $10^{-4}$.

In the image compression R-D curves, $\lambda$ ranges from $2^{-6}$ to $2^{16}$.
In Figure 5 of the main text, $\lambda$ was set to $17.5$ for VBQ to match the bitrate of the other methods. The uniform quantization result was obtained with 4 quantization levels, on a separately tuned model that had an additional convolutional layer of 64 channels. The additional conv layer was to reduce the latent dimensionality, as uniform quantization could not achieve bitrates lower than 0.5 even with only 2 grid points in the original 3-layer model.

\section{Additional Image Compression Examples}
\label{app:examples}

Starting on the next page, we provide detailed compression results for individual images from the Kodak dataset.
For each image, we show the rate-distortion performance by various methods, followed by reconstructions using our proposed method and JPEG at equal bitrate.%
\footnote{The present version of this document contains a subset of example images due to a file size limit on arXiv submissions.}

\newpage
\onecolumn

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/1.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/1.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/1.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.27-PSNR=23.595-MSSSIM=0.871.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.27, PSNR: 23.595, MS-SSIM: 0.871}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/1.png-jpg-quality=5-BPP=0.27-PSNR=22.015-MSSSIM=0.816.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.27, PSNR: 22.015, MS-SSIM: 0.816}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/2.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/2.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/2.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.19-PSNR=29.226-MSSSIM=0.882.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.19, PSNR: 29.226, MS-SSIM: 0.882}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/2.png-jpg-quality=7-BPP=0.19-PSNR=26.658-MSSSIM=0.747.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.19, PSNR: 26.658, MS-SSIM: 0.747}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/3.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/3.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/3.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.19-PSNR=30.664-MSSSIM=0.940.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.19, PSNR: 30.664, MS-SSIM: 0.94}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/3.png-jpg-quality=6-BPP=0.19-PSNR=26.201-MSSSIM=0.842.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.19, PSNR: 26.201, MS-SSIM: 0.842}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/8.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/8.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/8.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.34-PSNR=22.177-MSSSIM=0.903.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.34, PSNR: 22.177, MS-SSIM: 0.903}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/8.png-jpg-quality=5-BPP=0.34-PSNR=21.331-MSSSIM=0.882.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.34, PSNR: 21.331, MS-SSIM: 0.882}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/10.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/10.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}
 \begin{subfigure}[h]{0.49\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/imgs/10.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.22-PSNR=29.584-MSSSIM=0.935.png}\caption{Proposed. bits-per-pixel: 0.22, PSNR: 29.584, MS-SSIM: 0.935}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.49\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/imgs/10.png-jpg-quality=7-BPP=0.22-PSNR=26.543-MSSSIM=0.846.png}\caption{JPEG. bits-per-pixel: 0.22, PSNR: 26.543, MS-SSIM: 0.846}
 \end{subfigure}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/15.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/15.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/15.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.20-PSNR=29.523-MSSSIM=0.928.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.2, PSNR: 29.523, MS-SSIM: 0.928}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/15.png-jpg-quality=7-BPP=0.20-PSNR=26.600-MSSSIM=0.838.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.2, PSNR: 26.6, MS-SSIM: 0.838}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/16.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/16.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/16.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.21-PSNR=28.770-MSSSIM=0.908.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.21, PSNR: 28.77, MS-SSIM: 0.908}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/16.png-jpg-quality=7-BPP=0.21-PSNR=26.247-MSSSIM=0.818.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.21, PSNR: 26.247, MS-SSIM: 0.818}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/17.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/17.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}
 \begin{subfigure}[h]{0.49\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/imgs/17.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.20-PSNR=29.006-MSSSIM=0.936.png}\caption{Proposed. bits-per-pixel: 0.2, PSNR: 29.006, MS-SSIM: 0.936}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.49\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/imgs/17.png-jpg-quality=5-BPP=0.20-PSNR=25.389-MSSSIM=0.835.png}\caption{JPEG. bits-per-pixel: 0.2, PSNR: 25.389, MS-SSIM: 0.835}
 \end{subfigure}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/23.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/23.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/23.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.20-PSNR=31.425-MSSSIM=0.945.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.2, PSNR: 31.425, MS-SSIM: 0.945}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/23.png-jpg-quality=7-BPP=0.20-PSNR=26.976-MSSSIM=0.833.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.2, PSNR: 26.976, MS-SSIM: 0.833}
\end{figure}

\begin{figure}[t]
\centering
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/24.png-PSNR_RGB.pdf}
 \end{subfigure}
 \hfill
 \begin{subfigure}[h]{0.45\columnwidth}
 \centering
 \includegraphics[width=\textwidth]{figs/img_compression/kodak/RDs/24.png-MS-SSIM_RGB.pdf}
 \end{subfigure}

 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/24.png-ChannelwisePriorCDFQuantizer-max_bits=10-BPP=0.24-PSNR=24.254-MSSSIM=0.882.png}\vspace{-10pt}\caption{Proposed. bits-per-pixel: 0.24, PSNR: 24.254, MS-SSIM: 0.882}
 \vspace{10pt}\includegraphics[width=0.7\textwidth]{figs/img_compression/kodak/imgs/24.png-jpg-quality=5-BPP=0.24-PSNR=22.562-MSSSIM=0.818.png}\vspace{-10pt}\caption{JPEG. bits-per-pixel: 0.24, PSNR: 22.562, MS-SSIM: 0.818}
\end{figure}

\end{document}

redpajama_set_name: RedPajamaArXiv


text:
Subject: He Hasn't Taken a Loss on a Single Trade
URL: https://mney.co/2P1D1mF
1 + = 3 Required Please enter the correct value.
Follow Buy & Hold Stocks
Profit Alerts: Buy & Hold Stocks, Small Cap Stocks Alerts, Stocks to Watch, Trading Strategy Alerts
He Hasn't Taken a Loss on a Single Trade
By Money Morning Reports, Money Morning • October 1, 2018
As far as we know, he's the only undefeated individual in this entire industry.
If he has to scrutinize every number on every SEC filing that a company has ever made – until he feels comfortable enough betting his life that the stock will go up…
Then that's exactly what he'll do!
This isn't a side job or hobby for him. It's his livelihood.
He doesn't buy into hyperbolic nonsense.
He doesn't listen to 30-year-old MBAs who think that because they took a class in microeconomics, they know the future.
NO ONE – We don't care if you're a wizard with enchanted powers or Warren Buffett's third cousin…
NO ONE can predict the future price of a stock. If anyone tells you otherwise, IGNORE THEM.
There are no magic rings, flying monkeys, or mystical unicorns on stock charts that tell you when to buy and sell shares.
These are nothing but gimmicks crafted by lazy people who have ZERO talent, big egos, and the attention span of a four-year-old.
All he cares about is one thing – WINNING, pure and simple…
He's Extremely Picky About Where He Puts His Money
His undefeated status means EVERYTHING to him… he will never RISK anything to lose it.
So he treats every investment like a life-or-death situation.
At this point, nothing could be more harmful than making a losing trade recommendation.
Did You Miss This Epic Event? An undefeated strategist exposed his secret to 119 wins, 0 realized losses. Hurry – because if you don't watch the replay now, you may miss out forever…
That's why if he's not comfortable putting $1 million of his own money into a stock, he won't recommend touching it – not with a 10-foot pole.
And he's ONLY interested in applying his methodology to certain subsectors of the market – areas he's personally had a TON of success in, and though he doesn't have a crystal ball, he plans on keeping up the pace.
In case you didn't know, he has made 119 trade recommendations with zero realized losses.
Unless the world ends tomorrow, he sees no reason why that would ever change.
The question now is…
Do you want to join the ONLY UNDEFEATED team in the league?
Tags: Stocks
David Firmin | October 2, 2018
Long may he retain that undefeated position!!
Small Cap Stocks Alerts (this article)
Buy & Hold Stocks (this article)
Stocks to Watch (this article)

redpajama_set_name: RedPajamaCommonCrawl


text:
Sport AFL Sam Newman's 'hurtful' attack on retiring footy legend
4:23pm, Aug 25, 2017 Updated: 8:58pm, Aug 25
Sam Newman's 'hurtful' attack on retiring footy legend
Murphy and Easton Wood celebrate after the 2016 AFL Grand Final. Photo: Getty
James Willoughby
Sam Newman has been criticised for "distasteful" comments about retiring Western Bulldogs legend Robert Murphy and his role in the club's premiership celebrations last year.
Murphy missed last year's Grand Final win due to a serious knee injury as the Bulldogs broke a 62-year premiership drought in stunning fashion.
The 311-gamer, credited with turning the fortunes of the Bulldogs around in an inspirational stint as skipper, enjoyed the after-match celebrations with his teammates and even hoisted the premiership cup aloft with deputy captain Easton Wood to spark a big roar from the MCG crowd.
But nothing could make up for the fact Murphy did not play, as he has revealed many times this year, including an interview in which he said missing out left him with a "hole in the heart".
Murphy is set to retire at season's end and it is likely the Bulldogs will miss the finals, needing to beat Hawthorn and a slew of other results to go their way to feature in this season's top eight.
Speaking on The Footy Show, Newman said if the Bulldogs missed the finals "they won't have to go through the charade of finding a premiership medal for Bobby Murphy."
"Get out of the way, mate," he added.
"You've been in every photo of the premiership and actually didn't play in it.
"Let the players take the credit who actually won the thing."
Newman's opinion was not shared by The Footy Show host Eddie McGuire, who said Murphy "helped turn the culture of that football club around."
Former Essendon champion Tim Watson went on the attack on Friday morning, telling SEN radio "what Sam said last night is a really hurtful, distasteful thing to say."
Watson, who is a colleague of Murphy's at SEN, went on to say: "Bob Murphy is a genuine person, a most genuine person.
"He did nothing other than join in the celebration of the club's unlikely flag. He, more than most, understands the game is all about the team.
"The fact that he was dragged into the group was an acknowledgement of what he meant to the group and how deeply they felt for someone who had given so much, but missed the Grand Final day experience."
Watson added that the AFL was "littered" with people who "sought the spotlight" but that Murphy was not one of them.
"I'm angry about that [what Newman said] because I know it is not the way that he [Bob] would like to portray himself, it's not the way he wants to be portrayed by other people as well," he said.
"It was the emotion of the occasion that dragged him in."
The Last Goodbye: AFL prepares to farewell the 'class of 2017'


text:
Q: loginview change data source I have recently created an asp.net page in visual studio 2010.  I have not changed any of the prebuilt loginview code.  It creates the MDF file that allows for logging in users and registering users.  However, I don't want to use the MDF file, I want to use a SQL Server instance i have running (lets call it MyServer) with a database MyDatabase.  Where is the code that connects to the MDF file and is there a clean way of changing the data source?  Thanks!

A: You can modify your string connection in your config file (Web.config)
Modify this section
<connectionStrings>
   <add name="ConnectionString" connectionString="Data Source=.\SQLEXPRESS;AttachDbFilename=../File.mdf; Integrated Security=True;Connect Timeout=30;User Instance=True;Trusted_Connection=Yes"/>
</connectionStrings>

With for example this
<connectionStrings>
<add name="ConnectionString" connectionString="Data Source=myServerAddress;Initial Catalog=myDataBase;Integrated Security=SSPI;"/>
</connectionStrings>

Link : http://www.connectionstrings.com/

redpajama_set_name: RedPajamaStackExchange


Am I the only one, or do other parents out there in Parentville wish there was a place in between high school and college where our kids could go when they're not quite sure what they wanna be when they grow up? You know, someplace in between, where they could be exposed to a medley of professions, sort of like Dirty Jobs with Mike Rowe. Just maybe not quite as dirty. Cause I don't know about you, but I've always felt like it's a huuuuuge stretch to assume that our seventeen or eighteen-year-old kids have their future all figured out in time to pick their fall classes for freshman year.
I mean, sure, some kids know what they want to do with their lives from the time they're little and they follow that track in college and end up in that field as adults. And that's fantastic for them. But they're the minority, because I found a statistic in an MSNBC.com article that confirms that eighty percent of college-bound students have yet to choose a major. But they are still expected to pick schools and apply to and start degree programs without knowing where they want to end up. And that's staggering. Eighty. Percent.
What I'm talking about is separate from a gap year—aka a sabbatical—that's designed to let our kids take a step back between high school and college, so they can maturely consider the benefits of a higher-education. That's a different kind of option altogether. I'm suggesting something more like a series of internships, for lack of a better word, in their potential field(s) of study, so they can really see, first-hand, what a career in education or law or nursing is like from the inside and not just from a lecture hall.
Because with a daughter who just graduated from high school and who isn't one-hundred percent sure which field she wants to pursue, I know I'd love to see her be able to "try on" a few different careers before she buys, if you know what I mean? Cause I think we can all agree that not every kid is ready to jump straight into a field right after high school. Especially the kids who have zero idea what they want to do with their lives.
So, wouldn't it be nice if a place existed where our high-school-graduate kids could spend two months at a time, over a period of a traditional ten-month academic year, testing out four or five different career paths. Then, once they've been exposed to a handful of their top career choices, and gotten that three-dimensional chance to experience them, they can move on to college with a more concrete idea of what direction to take when they finally do declare their major.
Lisa Sugarman lives just north of Boston, Massachusetts. Read and discuss all her columns at lisasugarman.com. Or, find them on GrownAndFlown.com, Hot Moms Club, LittleThings.com, MommingHub.com, More Content Now, Wickedlocal.com, and Care.com. She is also the author of LIFE: It Is What It Is and Untying Parent Anxiety: 18 Myths That Have You in Knots—And How to Get Free available on Amazon, at Barnes & Noble, and at select bookstores everywhere.


text:
package util;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Properties;

public class SettingWriter {
    private final static String L1GHTS4J_PROPS = "l1ghts4j.properties";

    private final static String UPDATE_NAME_ACCESS_LEVEL = "updateName.accessLevel";
    private final static String UPDATE_LOCATION_ACCESS_LEVEL = "updateLocation.accessLevel";
    private final static String UPDATE_ICON_ACCESS_LEVEL = "updateIcon.accessLevel";

    private static void Writer(String FileName, String targetName, String value) {
        Properties prop = new Properties();

        try {
            prop.load(new FileInputStream(FileName));
        } catch (FileNotFoundException e) {
            System.out.println("[ERROR] " + FileName + " does not found.");
            e.printStackTrace();
        } catch (IOException e) {
            System.out.println("[ERROR] IOException has occurred.");
            e.printStackTrace();
        }
        prop.setProperty(targetName, value);
    }

    /**
     * UpdateNameのアクセスレベルをセットするクラス
     */
    public static void setUpdateNameAccessLevel(String setValue) {
        if (setValue.equals("0") || setValue.equals("1") || setValue.equals("2")) {
            Writer(L1GHTS4J_PROPS, UPDATE_NAME_ACCESS_LEVEL, setValue);
        } else {
            throw new IllegalArgumentException(String.format("%s is incorrect value.", setValue));
        }
    }

    /**
     * UpdateLocationのアクセスレベルをセットするクラス
     */
    public static void setUpdateLocationAccessLevel(String setValue) {
        if (setValue.equals("0") || setValue.equals("1") || setValue.equals("2")) {
            Writer(L1GHTS4J_PROPS, UPDATE_LOCATION_ACCESS_LEVEL, setValue);
        } else {
            throw new IllegalArgumentException(String.format("%s is incorrect value.", setValue));
        }
    }

    /**
     * UpdateIconのアクセスレベルをセットするクラス
     */
    public static void getUpdateIconAccessLevel(String setValue) {
        if (setValue.equals("0") || setValue.equals("1") || setValue.equals("2")) {
            Writer(L1GHTS4J_PROPS, UPDATE_ICON_ACCESS_LEVEL, setValue);
        } else {
            throw new IllegalArgumentException(String.format("%s is incorrect value.", setValue));
        }
    }

}

redpajama_set_name: RedPajamaGithub


text:
Q: Uploading/Arduino gets an error : permission denied on Ubuntu 16.04 I have faced this problem for few years ago and I solved it on Ubuntu 14.04 but I don't know why my solution doesn't work right now on Ubuntu 16.04
The Error:
avrdude: ser_open(): can't open device "/dev/ttyACM0": Permission denied

I'm using the same solution from The guide , by searching for solution on Stack sites and more ...
Here what I've done :
ls -l /dev/ttyACM*

from the result : The owner is root and the
group with access is dialout
Then I applied:
sudo usermod -a -G dialout <username>
sudo chmod a+rw /dev/ttyACM0

Finally logged out and logged in.
Sompe people said they've solved it by removing modemmanager so I have tried that as well :
sudo apt-get remove modemmanager

But after all , the problem hasn't solved yet
Note: Arduino Uno with Ubuntu 16.04
Edited:
After apply :
ls -l /dev/ttyACM*

The result is :
crw-rw---- 1 root dialout 166, 0 Jan 10 13:33 /dev/ttyACM0

A: The problem is (as you already figured out), that your user is not allowed to access the /dev/ttyACM0 device.
You have several possibilities:

*

*temporarily fix the issue for your current session
sudo chown <user>:<group> /dev/ttyACM0

This changes the ownership of the file to user and the group to group
(note: you will have to do this again, as soon as you replug your arduino)

*permanently fix the issue for each session
you create a udev-rule for your arduino and put it into /etc/udev/rules.d (you should be able to find a correct udev-rule for your arduino when googling for it).
The trivial approach would be the following udev-rule:
99-arduino.rules
KERNEL=="ttyACM0",GROUP="plugdev"

please be aware that now every /dev/ttyACM0 device will be owned by the group "plugdev" (or whatever group you specify). This might not be, what you really want. So please read into udev-rules how to specify your arduino-device exclusively (hint: vendorID and productID)


Why is The Angel Company Business Model Superior?
Now that the final crowdfunding rules for SEC Title III as well as Reg A+ have been published we know what private companies need to do in order for them to initially offer their equity to non-accredited investors.
The Angel Company has a patented system and method that will make it dominate the coming private equity trading market.
These patents are enormously valuable. Only the patent holder and its licensees are legally allowed to perform these practices.
Create and trade the equivalent of mutual funds of private equity companies.
© The Angel Company. All rights reserved.


I'm bored, so I thought I'd share with wewe a few websites online that wewe can ubunifu your characters! There are tons of websites out there, these are just a few.
This website is kwa far the best superhero generator I've found. Basically wewe use it to make your character in costume. They have endless options of costume parts for both male and female characters, and they even allow wewe to songesha parts around, resize them, and color them. wewe can save the character to the game, and then load it later, but if wewe want to save an image to the computer, wewe have to take a screenshot.
There are lots and lots of anime maker games on Deviantart, but this one is kwa far the best, and one of the cutest, in my opinion. It has an almost overwhelming amount of hair choices, and other face options. Since the Avatar is from the shoulders up, wewe only see the juu part of the clothes, but they still offer a wide range of clothes options. wewe can do a boy au a girl (the only thing that has seperate items for different genders is the clothes) and the only way to save is kwa taking a screenshot.
wewe probably know that there are tons of dress-up game websites out there, but I find this one to be kwa far the best. They have lots and lots of games to play, some anime style, others not so much, that vary in the amount of options and items wewe can use. Some of my inayopendelewa games are the Fashion Creator Dress Up (where wewe can ubunifu an outfit), the Shoujo manga Avatar creators (limited options, but really cute) and the link (there are 11 pages in all, most of them are on the romantic side). This website does have a save option, which automatically saves an image to your computer.
This makes cute, tiny, chibi-style characters that wewe can save kwa taking a screenshot au kwa saving it to the website. There are lots of clothing options, so many in fact that it's almost impossible to find what wewe need kwa browsing, so I recommend that wewe use the tafuta bar au narrow the orodha kwa color au item type.
This is a super cute little chibi creator. There is a huge range of options for hair, face, outfits (mostly dresses and anime-type clothes but there are literally thousands of things to choose from and lots of 'boy clothes' and stuff also), and accessories/backgrounds. The tafuta feature isn't all that great, however, but it's endless fun to look through the different categories and pick cute stuff (you can sort kwa theme-category, color, type [shirt, hairstyle, etc.], au search). To save an image to your computer you'll have to screenshot but the site gives wewe the option to save a link which wewe can bookmark on your computer and access anytime, and a window to view and screenshot a big au small image of your chibi.
So that's all I've got. If wewe guys know of any other good websites, let me know and I'll add them to this list! Bai now!


37th Annual Best Ball Tournament | $8650+ in Payouts!
Join us for a long weekend of great fun and friends. The 37th Annual Best Ball Tournament is always a full field and a perfect golf weekend.
Included: Three rounds of golf, range balls, lunch on Saturday and Sunday, and tee prizes!
Eligibility: Max handicap: 34.0 - Using 90% for competition. Partners must be within eight (8) handicap strokes.


text:
F1: Nico Rosberg wins the First GP of Europe in Baku
Nico Rosberg rebuilds Formula 1 points lead with dominant Baku win. His title rival, Hamilton, finished back in fifth.
The F1 carnival had a super fast change-over last week, having to transport all 11 teams from Montreal, Canada all the way over to Azerbaijan, in the far reaches of Eastern Europe, for the fist ever GP of Europe.
For Nico Rosberg, the trip was well worth it; after a frustrating weekend for his Mercedes team-mate, Lewis Hamilton, he extended his world championship lead with a crushing victory. After securing pole position in qualifying, the German led into the first corner and set about building a gap between the chasing Red Bull of Daniel Ricciardo. In the end, it was a gap which just grew and grew throughout the race and the cherry on top of Rosberg's cake was that Hamilton, his title rival, finished back in fifth.
It was Rosberg's fifth victory of the season and first since Russia, leaving him 24 points clear of Hamilton in the drivers' standings.
Having crashed out in Q3, Hamilton made good progress through the field from 10th, but then encountered lower than maximum electrical deployment from his hybrid engine. He repeatedly asked his engineer for help in solving the problem but Mercedes were unable to give any substantial guidance, due to this season's radio communication restrictions, and were only able to advise that is was a problem with one of the engine modes.
Hamilton was able to fix the problem in the closing stages but it proved to be too late and he was unable to improve on fifth.
"The radio ban, as far I'm aware, was supposed to stop driver aids," said Hamilton. He continued:
That wasn't a driver aid, that was a technical issue. The FIA know that Formula 1 is so technical, far too technical almost. To have that many switch positions, that's something you should be able to rectify because the only people who can see the issue are the guys in the garage. Today, it would've added to the spectacle if I would have had full power because I would've been more in the race, fighting with the guys up ahead.
Ferrari's Sebastian Vettel finished second for the second successive race with Force India's Sergio Perez scoring his second podium in three races with third. The Mexican passed Kimi Raikkonen on the final lap to snatch third, but he was set to get the place anyway as Raikkonen had been given a five-second time penalty for crossing the white line on pit entry earlier in the race.
Perez's team-mate Hulkenberg finished ninth, one place ahead of Felipe Massa, who scored points for the seventh time in eight races. This time last year he won the 24h of the Mans with Porsche, who snatched the victory from Toyota on the last lap, for a successive victory this year.
1 Nico Rosberg Mercedes 1h32m52.366s
2 Sebastian Vettel Ferrari 16.696s
3 Sergio Perez Force India/Mercedes 25.241s
4 Kimi Raikkonen Ferrari 33.102s
5 Lewis Hamilton Mercedes 56.335s
6 Valtteri Bottas Williams/Mercedes 1m00.886s
7 Daniel Ricciardo Red Bull/Renault 1m09.229s
8 Max Verstappen Red Bull/Renault 1m10.696s
9 Nico Hulkenberg Force India/Mercedes 1m17.708s
10 Felipe Massa Williams/Mercedes 1m25.375s
11 Jenson Button McLaren/Honda 1m44.817s
12 Felipe Nasr Sauber/Ferrari 1 Lap
13 Romain Grosjean Haas/Ferrari 1 Lap
14 Kevin Magnussen Renault 1 Lap
15 Jolyon Palmer Renault 1 Lap
16 Esteban Gutierrez Haas/Ferrari 1 Lap
17 Marcus Ericsson Sauber/Ferrari 1 Lap
18 Rio Haryanto Manor/Mercedes 2 Laps
– Fernando Alonso McLaren/Honda Gearbox
– Pascal Wehrlein Manor/Mercedes Brakes
– Carlos Sainz Toro Rosso/Ferrari Suspension
– Daniil Kvyat
Featured image: Getty Images / Red Bull Content Pool
Posted in: Motorsports Tagged: Formula 1| Motorsports
The street racing lifestyle's appeal and potential dangers
Keep the racing to the track.
The complete motocross helmet buyer's guide
Here's everything you should look for in a new motocross helmet.
Insane video shows dirt biker crashing 100 feet down cliff into waterfall at Colorado's Schofield Pass
Incredibly the rider was completely uninjured and climbed out within 15 minutes of the incident.
'Every racing driver will need a sim racing profile to reach the top' says Team Redline boss
The lines between 'real-world' and sim racing are getting blurred; Team Redline manager Dom Duhan thinks that this trend is set to continue.
Sim racing officially recognised as a motorsports discipline in Germany
Germany's motorsport governing body Deutsche Motor Sport Bund sees a close relationship between real-world racing and sim racing.
McLaren F1 team cut partnership with Honda for 2018
It's been circling around the F1 paddock for weeks, but now it's confirmed. McLaren will be dropping their Japanese engine supplier.
Watch the moment an Isle of Man TT racer pulls off the most unbelievable save to avoid crashing
James 'Dynamite' Hillier narrowly avoided crashing at 150 mph during the opening lap of the Pokerstars Senior Race.
F1 drivers spotted white water rafting ahead of this weekend's Canadian Grand Prix
Ricciardo, Verstappen, Kvyat and Sainz all took to white water rafts to tackle the chopping waters.
A Barcelona political party is calling for the end of grid girls in MotoGP
If successful, the ruling would also apply to all events held at the Barcelona track, including future F1 Spanish Grands Prix.
Guardrail saves rally driver from tumbling off cliff
The railing held the car as the driver and co-driver escaped to safety.
McLaren launch gaming competition to find next Formula 1 simulator driver
McLaren has launched an esports competition, with new sponsor Logitech, to find the best sim racing gamer in the world.
Japanese rider smashes world record for continuous wheelie after 13 hours on a scooter
Masaru Abe rode for 500km, that's the same distance as London to Cologne.