text:
\subsubsection*{Acknowledgements}
We would like to thank Murray Shanahan and Nick Watters for useful feedback that improved the manuscript.
\section{Conclusion}
This paper introduced a new approach to learning grounded visual concepts. We defined concepts as abstractions over independent (and often interpretable) visual primitives, where each concept is given by learned distributions over a set of relevant visual factors. We proposed that all other (irrelevant) visual factors should default to their prior in order to produce a diverse set of samples corresponding to a concept. We then proposed SCAN\xspace, a neural network implementation of such an approach, which was able to discover and learn an implicit hierarchy of abstract concepts from as few as five symbol-image pairs per concept and no assumptions on the nature of symbolic representations. SCAN\xspace was then capable of bi-directional inference, generating diverse and accurate image samples from symbolic instructions, and vice versa, qualitatively and quantitatively outperforming all baselines, including on a realistic CelebA dataset with noisy attribute labels. The structure of the learnt concepts allowed us to train an extension to SCAN\xspace that could perform logical recombination operators. We demonstrated how such operators could be used to traverse the implicit concept hierarchy, including imagining completely new concepts. Due to the sample efficiency and the limited number of assumptions in our approach, the representations learnt by SCAN\xspace should be immediately applicable within a large set of broader problem domains, including reinforcement learning, classification, control and planning.

\section{Experiments}
\label{sec_experiments}
\subsection{DeepMind Lab experiments}

\paragraph{Environment}
We evaluate the performance of SCAN\xspace on a dataset of visual frames and corresponding symbolic descriptions collected within the DeepMind Lab environment \citep{deepmind_lab}. DeepMind Lab was chosen, because it gave us good control of the ground truth generative process. The visual frames were collected from a static viewpoint situated in a room containing a single object. The generative process was specified by four factors of variation: wall colour, floor colour, object colour with $16$ possible values each, and object identity with $3$ possible values: hat, ice lolly and suitcase. Other factors of variation were also added to the dataset by the DeepMind Lab engine, such as the spawn animation, horizontal camera rotation and the rotation of objects around the vertical axis. We split the dataset into two subsets. One was used for training the models, while the other one contained a held out set of $300$ four-gram concepts that were never seen during training, either visually or symbolically. We used the held out set to evaluate the model's ability to imagine new concepts.
\begin{figure}[t!]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/img2sym_sym2img_compress_v4.png}
 \vspace{-6pt}
 \caption{\textbf{A}: sym2img inferences with \sym{white suitcase}, \sym{white suitcase, blue wall}, and \sym{white suitcase, blue wall, magenta floor} as input. The latter one points to a concept that the model has never seen during training, either visually or symbolically. All samples are consistently accurate, while showing good diversity in terms of the irrelevant visual attributes. \textbf{B}: when presented with an image, SCAN\xspace is able to describe it in terms of all concepts it has learnt, including synonyms (e.g. \sym{dub}, which corresponds to \concept{ice lolly, white wall}). The histograms show the distributions of unique concepts the model used to describe each image, most probable of which are printed in descending order next to the corresponding image. The few confusions SCAN\xspace makes are intuitive to humans too (e.g. confusing orange and yellow colours).}
 \label{fig_bi_inference_single_concepts}
\end{figure}

\paragraph{Learning grounded concepts}
In this section we demonstrate that SCAN\xspace is capable of learning the meaning of new concepts from very few image-symbol pairs. We evaluate the model's concept understanding through qualitative analysis of sym2img and img2sym samples. First we pre-trained a \ensuremath{\beta}-VAE\xspace to learn a disentangled representation of the DeepMind Lab dataset (see Sec.\ \ref{sup_baselines} in Supplemenrary Materials for details). Then we trained SCAN\xspace on a random subset of $133$ out of $18,883$ possible concepts sampled from all levels of the implicit hierarchy (these concepts specify between one and four visual primitives, and are associated with 1- to 4-hot symbols respectively). The set of symbols also included a number of 1-hot synonyms (e.g. a blue wall may be described by symbols \sym{blue wall}, \sym{bright blue wall} or \sym{blue wall synonym}). Each concept was associated with ten visual examples during training.

Fig.~\ref{fig_bi_inference_single_concepts}A shows samples drawn from SCAN\xspace when asked to imagine a bigram concept \concept{white, suitcase}, a trigram concept \concept{white, suitcase, blue wall}, or a four-gram \concept{white, suitcase, blue wall, magenta floor}. Note that the latter is a concept drawn from the held-out test set that neither \ensuremath{\beta}-VAE\xspace nor SCAN\xspace have ever seen during training, and the first two concepts are novel to SCAN\xspace, but have been experienced by \ensuremath{\beta}-VAE\xspace. It is evident that the model demonstrates a good understanding of all three concepts, producing visual samples that match the meaning of the concept, and showing good variability over the irrelevant factors. Confusions do sometimes arise due to the sampling process (e.g. one of the suitcase samples is actually an ice lolly). Fig.~\ref{fig_bi_inference_single_concepts}B demonstrates that the same model can also correctly describe an image. The labels are mostly consistent with the image and display good diversity (SCAN\xspace is able to describe the same image using different symbols including synonyms). The few confusions that SCAN\xspace does make are between concepts that are easily confusable for human too (e.g. red, orange and yellow colours).

\paragraph{Evolution of concept understanding}
In this section we take a closer look inside SCAN\xspace as it learns a new concept. In Sec.~\ref{sec_formalising} we suggested that concepts should be grounded in terms of specified factors (the corresponding latent units $z_y^k\ \forall k \in S$ should have low inferred standard deviations $\sigma_y^k$), while the unspecified visual primitives should be sampled from the unit Gaussian prior (the corresponding latent units $z_y^k\ \forall k \in \overline{S}$ show have $\sigma_y^k \approx 1$). We visualise this process by teaching SCAN\xspace the meaning of the concept \concept{cyan wall} using a curriculum of fifteen progressively more diverse visual examples (see Fig.~\ref{fig_concept_evolution}, bottom row). After training SCAN\xspace on each set of five visual examples, we test the model's understanding of the concept through sym2img sampling using the symbol \sym{cyan wall} (Fig.~\ref{fig_concept_evolution}, top four rows). We also plot the average inferred specificity of all $32$ latent units $z_y^k$ during training (Fig.~\ref{fig_concept_evolution}, right). It can be seen that the number of specified latents $z_y^k$ drops from six, over four, to two as the diversity of visual examples seen by SCAN\xspace increases. The remaining two highly specified latents $z_y^k$ correctly correspond to the visual primitives $\myvec{z}_x$ representing wall hue and brightness.
\begin{figure}[t!]
 \centering

 \includegraphics[width = 1\textwidth]{figures/concept_learning_progression_v4.png}
 \vspace{-6pt}
 \caption{Evolution of understanding of the meaning of concept \concept{cyan wall} as SCAN\xspace is exposed to progressively more diverse visual examples. \textbf{Left}: top row contains three sets of visual samples (sym2img) generated by SCAN\xspace after seeing each set of five visual examples presented in the bottom row. \textbf{Right}: average inferred specificity of concept latents $z_y^k$ during training. Vertical dashed lines correspond to the vertical dashed lines in the left plot and indicate a switch to the next set of five more diverse visual examples. 6/32 latents $z_y^k$ and labelled according to their corresponding visual primitives in $\myvec{z}_x$.}
 \label{fig_concept_evolution}
\end{figure}

\paragraph{Quantitative comparison to baselines}
\label{sec_quant}
In this section we quantitatively compare the accuracy and diversity of the sym2img samples produced by the SCAN\xspace to those of the baselines -- a SCAN\xspace like architecture trained with a reverse KL used for grounding conceptual representations in vision (\ensuremath{\text{SCAN}_\text{R}}\xspace), another modification of SCAN\xspace that tries to ground conceptual representations in unstructured (entangled) visual representations (\ensuremath{\text{SCAN}_\text{U}}\xspace, with various levels of visual entanglement), and two of the latest multimodal joint density models, the JMVAE~\citep{Suzuki_etal_2017} and the triple ELBO (TrELBO)~\citep{Vedantam_etal_2017}. The two metrics, accuracy and diversity, measure different aspects of the models' performance. High accuracy means that the models understand the meaning of a symbol (e.g. samples of a \sym{blue suitcase} should contain blue suitcases). High diversity means that the models were able to learn an abstraction. It quantifies the variety of samples in terms of the unspecified visual attributes (e.g. samples of blue suitcases should include a high diversity of wall colours and floor colours). There is a correlation between the two metrics, since samples with low accuracy often result in higher diversity scores.

We use a pre-trained classifier achieving $99\%$ average accuracy over all data generative factors to evaluate the accuracy of img2sym samples. Since some colours in the dataset are hard to differentiate even to humans (e.g. yellow and orange), we use top-3 accuracy for colour related factors. We evaluate the diversity of visual samples by estimating the KL divergence of the inferred factor distribution with the flat prior: $D_{KL}\divx{u(\myvec{y}_i)}{p(\myvec{y}_i)}$, where $p(\myvec{y}_i)$ is the joint distribution over the factors irrelevant to the $i$th concept $i \in \overline{S_{i}}$  (inferred by the classifier) and $u(\myvec{y}_i)$ is the equivalent flat distribution (i.e., with each factor value having equal probability). See Sec.~\ref{sup_acc_div_details} in Supplementary Materials for more details.

All models were trained on a random subset of $133$ out of $18,883$ possible concepts sampled from all levels of the implicit hierarchy with ten visual examples each. The accuracy and diversity metrics were calculated on two sets of sym2img samples: 1) \emph{train}, corresponding to the $133$ symbols used to train the models; and 2) \emph{test (symbols)}, corresponding to a held out set of $50$ symbols. Tbl.~\ref{tbl_quant_results} demonstrates that SCAN\xspace outperforms all baselines in terms of both metrics. \ensuremath{\text{SCAN}_\text{R}}\xspace learns very accurate representations, however it overfits to a single mode of each of the irrelevant visual factors and hence lacks diversity. \ensuremath{\text{SCAN}_\text{U}}\xspace experiments show that as the level of disentanglement within the visual representation is increased (the higher the $\beta$, the more disentangled the representation), the accuracy and the diversity of the sym2img samples also get better. Note that baselines with poor sample accuracy inadvertently have good diversity scores because samples that are hard to classify produce a relatively flat classifier distribution $p(\myvec{y}_i)$ close to the uniform prior $u(\myvec{y}_i)$. TrELBO learns an entangled and unstructured conceptual representation that produces accurate yet stereotypical sym2img samples that lack diversity. Finally, JMVAE is a model that comes the closest to SCAN\xspace in terms of performance. It manages to exploit the structure of the symbolic inputs to learn a representation of the joint posterior that is almost as disentangled as that of SCAN\xspace. Similarly to SCAN\xspace, it also uses a forward KL term to match unimodal posteriors to the joint posterior. Hence, given that there is enough supervision within the symbols to help JMVAE learn a disentangled joint posterior, it should become equivalent to SCAN\xspace, whereby the joint $q(\myvec{z}|\myvec{x}, \myvec{y})$ and unimodal $q(\myvec{z}|\myvec{y})$ posteriors of JMVAE become equivalent to the visual $q(\myvec{z}_x|\myvec{x})$ and symbolic $q(\myvec{z}_y|\myvec{y})$ posteriors of SCAN\xspace respectively. Yet in practice we found that JMVAE training is much more sensitive to various architectural and hyperparameter choices compared to SCAN\xspace, which often results in mode collapse leading to the reasonable accuracy yet poor diversity of the JMVAE sym2img samples. See Sec.~\ref{sup_baselines} for more details of the baselines' performance. Finally, SCAN\xspace is the only model that was able to exploit the k-hot structure of the symbols and the compositional nature of its representations to generalise well to the test set (\emph{test symbols} results), while all of the other baselines lost a lot of their sample accuracy.

\begin{table}[t!]
    \begin{center}
    \begin{small}
    \vspace{-12pt}
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{lccc||ccc}
        \hline
        & \multicolumn{3}{c||}{\textsc{\textbf{Accuracy}}} & \multicolumn{3}{c}{\textsc{\textbf{Diversity}}} \\
        \textsc{Model}       &  \textsc{Train} & \textsc{Test (symbols)} & \textsc{Test (operators)}  & \textsc{Train} & \textsc{Test (symbols)} & \textsc{Test (operators)} \\
        \hline
        \textsc{TrELBO}               &  0.81 & 0.69 & 0.37 & 9.41 & 6.86 & \textbf{0.63} \\
        \textsc{JMVAE}               &  0.75  &   0.68 & 0.61 & 4.32 & 2.87 & 0.86  \\
        \textsc{\ensuremath{\text{SCAN}_\text{R}}\xspace}         &  \textbf{0.86}  &  \textbf{0.81} & 0.67 & 13.17 & 9.2 & 9.94  \\
        \textsc{\ensuremath{\text{SCAN}_\text{U}}\xspace $(\beta=0.1)$}  &  0.27 & 0.26 & 0.25 & 5.51 & 1.23 & 1.66 \\
        \textsc{\ensuremath{\text{SCAN}_\text{U}}\xspace $(\beta=1)$}  &  0.58 & 0.36 & 0.33 & 2.07 & 1.22 & 1.34 \\
        \textsc{\ensuremath{\text{SCAN}_\text{U}}\xspace $(\beta=20)$}  &  0.65 & 0.42 & 0.32 & \textbf{1.41} & 3.98 & 4.57 \\
        \hline
        \textsc{\textbf{SCAN} $(\beta=53)$}   & 0.82 & 0.79 & \textbf{0.79} & 1.46 & \textbf{1.08} & 1.05 \\
        \hline
        \hline
        \end{tabular}
    }
    \vspace{-5pt}
    \caption{Quantitative results comparing the accuracy and diversity of visual samples produced through sym2img inference by SCAN\xspace and three baselines: a SCAN\xspace with unstructured vision (\ensuremath{\text{SCAN}_\text{U}}\xspace, lower $\beta$ means more visual entanglement), a SCAN\xspace with a reverse grounding KL term for both the model itself and its recombination operator (\ensuremath{\text{SCAN}_\text{R}}\xspace) and two recent joint multimodal embedding models, JMVAE and TrELBO. Higher accuracy and lower diversity indicate better performance. Test values can be computed either by directly feeding the ground truth symbols (\emph{test symbols}), or by applying trained recombination operators to make the model recombine in the latent space (\emph{test operators}).}
    \label{tbl_quant_results}
    \end{small}
    \end{center}
\vspace{-10pt}
\end{table}

\paragraph{Learning recombination operators}
In Sec.~\ref{sec_recombinations} we suggested a way to traverse the implicit hierarchy of concepts towards novel nodes without any knowledge of how to point to these nodes through a symbolic reference. We suggested doing so by instructing a recombination of known concepts in the latent space. To test this, we trained a recombination module using $10$ recombination instructions per each of the three operators, with $20$ visual examples each. Tbl.~\ref{tbl_quant_results} (\emph{test operators}) demonstrates that we were able to reach the nodes corresponding to the $50$ novel test concepts using such a pre-trained recombination operator module. This, however, only worked for SCAN\xspace, since the successful training of the recombination module relies on a structured latent space that all the other baselines lacked. SCAN\xspace with the recombination module preserved the accuracy and the diversity of samples, as shown quantitatively in Tbl.~\ref{tbl_quant_results} and qualitatively in Fig.~\ref{fig_imagining_recombinations}B. JMVAE, the closest baseline to SCAN\xspace in terms of the recombination module performance, produced samples with low accuracy (the drop in accuracy resulted in an increase in the diversity score). It is interesting to note that the recombination operator training relies on the same kind of visual grounding as SCAN, hence it can often improve the diversity of the original model.

\subsection{CelebA experiments}
We ran additional experiments on a more realistic dataset CelebA \citep{Liu_etal_2015} after performing minimal dataset pre-processing of cropping the frames to 64x64. Unlike other approaches \citep{Vedantam_etal_2017, Perarnau_etal_2016} which only use 18 best attributes for training their models, we used all 40 attributes. Many of these 40 attributes are not useful, since they are either: 1) subjective (e.g. \sym{attractiveness}); 2) refer to parts of the image that have been cropped out (e.g. \sym{wearing necktie}); 3) refer to visual features that have not been discovered by \ensuremath{\beta}-VAE\xspace (e.g. \sym{sideburns}, see \cite{Higgins_etal_2017} for a discussion of the types of factors that \ensuremath{\beta}-VAE\xspace tends to learn on this dataset); 4) are confusing due to mislabelling (e.g. \sym{bald female} as reported by \cite{Vedantam_etal_2017}). Hence, our experiments test the robustness of SCAN\xspace to learning concepts in an adversarial setting, where the model is taught concepts that do not necessarily relate well to their corresponding visual examples. For these experiments we used the controlled capacity schedule \citep{Burgess_etal_2017} for \ensuremath{\beta}-VAE\xspace training to increase the quality of the generative process of the model.

We found that SCAN\xspace trained on CelebA was able to outperform its baselines of JMVAE and TrELBO. First, we checked which of the 40 attributes SCAN\xspace was able to understand after training. To do so, we inferred $q(\myvec{z}_y|\myvec{y}_i)$ for all $\myvec{y}_i \in \mathcal{R}^{40}$, where $\myvec{y}_i$ is a 1-hot encoding of the ith attribute. We then approximated the number of \emph{specified} latents for each posterior $q(\myvec{z}_y|\myvec{y}_i)$. If an attribute $i$ did not correspond to anything meaningful in the corresponding visual examples seen during training, it would have no \emph{specified} latents and $D_{KL}(q(\myvec{z}_y|\myvec{y}_i) || p(\myvec{z}_y) ) \approx 0$. We found that SCAN\xspace did indeed learn the meaning of a large number of attributes. Fig.~\ref{fig_celeba_baselines_attributes} shows \emph{sym2img} samples for some of them compared to the equivalent samples for the baseline models: JMVAE and TrELBO. It can be seen that SCAN\xspace samples tend to be more faithful than those of JMVAE, and both models produce much better diversity of samples than TrELBO.

\begin{figure}[t!]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/celeba_baselines_attributes.png}
 \vspace{-6pt}
 \caption{Comparison of sym2img samples of SCAN\xspace, JMVAE and TrELBO trained on CelebA. See Fig.~\ref{fig_celeba_baselines_attributes_large} in Supplementary Materials for larger samples.}
 \label{fig_celeba_baselines_attributes}
\end{figure}

A notable difference between SCAN\xspace and the two baselines is that despite being trained on \emph{binary} k-hot attribute vectors (where k varies for each sample), SCAN\xspace learnt meaningful directions of \emph{continuous} variability in its conceptual latent space $\myvec{z}_y$. For example, if we vary the value of an individual symbolic attribute, we will get meaningful sym2img samples that range between extreme positive and extreme negative examples of that attribute (e.g. by changing the values of the \sym{pale skin} symbol $\myvec{y}$, we can generate samples with various skin tones as shown in Fig.~\ref{fig_celeba_scan_traversals}). This is in contrast to JMVAE and TrELBO, which only produce meaningful sym2img samples if the value of the attribute is set to 1 (attribute is present) or 0 (attribute is not enforced). This means that unlike SCAN\xspace, it is impossible to enforce JMVAE or TrELBO to generate samples with darker skin colours despite the models knowing the meaning of the \sym{pale skin} attribute.

\begin{figure}[t!]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/celeba_scan_traversals.png}
 \vspace{-6pt}
 \caption{Example sym2img samples of SCAN\xspace trained on CelebA. We run inference using four different values for each attribute. We found that the model was more sensitive to changes in values in the positive rather than negative direction, hence we use the following values: $\{-6, -3, 1, 2\}$. See Fig.~\ref{fig_celeba_scan_traversals_large} in Supplementary Materials for larger samples.}
 \label{fig_celeba_scan_traversals}
\end{figure}

Note that sometimes SCAN\xspace picks up implicit biases in the dataset. For example, after training SCAN\xspace interprets \sym{attractive} as a term that refers to young white females and less so to males, especially if these males are also older and have darker skin tones (Fig.~\ref{fig_celeba_scan_traversals}). Similarly, SCAN\xspace learns to use the term \sym{big lips} to describe younger ethnic individuals, and less so older white males; while \sym{arched eyebrows} is deemed appropriate to use when describing young white females, but not when describing people wearing sunglasses or hats, presumably because one cannot see how arched their eyebrows are.

\section{Formalising concepts}
\label{sec_formalising}
In Sec.~\ref{sec_intro} we informally proposed that concepts are abstractions over visual representational primitives. Hence, in order to formally define concepts we first define the visual representations used to ground them. These are defined as tuples of the form $(Z_1, ..., Z_K)$, where $\{1, ..., K\}$ is the set of indices of the independent latent factors sufficient to generate the visual input $\myvec{x}$, and $Z_k$ is a random variable. The set $\mathbb{R}^K$ of all such tuples is a K-dimensional visual representation space.

We define a concept $C_i$ in such a K-dimensional representation space as a set of assignments of probability distributions to the random variables $Z_k$, with the following form:
\begin{align} \label{eq_concept}
    C_i = \{ (k, p_k^i(Z_k)) ~|~ k \in S_{i}  \}
\end{align}

where $S_{i} \subseteq \{1, ..., K\}$ is the set of visual latent primitives that are relevant to concept $C_i$ and $p_k^i(Z_k)$ is a probability distribution specified for the visual latent factor represented by the random variable $Z_k$. Since $S_i$ are subsets of $\{1, ..., K\}$, concepts are abstractions over the K-dimensional visual representation space. To generate a visual sample corresponding to a concept $C_i$, it is necessary to fill in details for latents that got abstracted away during concept learning. This corresponds to the probability distributions $\{p_k(Z_k) | k \in \overline{S_{i}}\}$, where $\overline{S_{i}} =  \{1, ..., K\} \setminus S_{i}$ is the set of visual latent primitives that are irrelevant to the concept $C_i$. In SCAN\xspace we set these to the unit Gaussian prior: $p_k(Z_k) = \mathcal{N}(0, 1), \; \forall k \in \overline{S_{i}}$.

In order to improve readability, we will use a simplified notation for concepts throughout the rest of the paper. For example, $\{\ (size,\ \  p(Z_{size} = \text{small})),\ \  (colour,\ \ p(Z_{colour} = \text{blue}))\ \}$ will become either \concept{small, blue} or \concept{small, blue, \_, \_}, depending on whether we signify the irrelevant primitives $k \in \overline{S_{i}}$ as placeholder symbols \enquote{\_}. Note that unlike the formal notation, the ordering of attributes within the simplified notation is fixed and meaningful.

Since we define concepts as sets, we can also define binary relations and operators on these sets. If $C_1$ and $C_2$ are concepts, and $C_1 \subset C_2$, we say that $C_1$ is \emph{superordinate} to $C_2$, and $C_2$ is \emph{subordinate} to $C_1$. Two concepts $C_1$ and $C_2$ are \emph{orthogonal} if $S_1 \cap S_2 = \varnothing$. The \emph{conjunction} of two orthogonal concepts $C_1$ and $C_2$ is the concept $C_1 \cup C_2$ (e.g. \concept{small, \_, \_} \operator{AND} \concept{\_, round, \_} = \concept{small, round, \_}). The \emph{overlap} of two non-orthogonal concepts $C_1$ and $C_2$ is the concept $C_1 \cap C_2$ (e.g. \concept{small, round, \_} \operator{IN COMMON} \concept{\_, round, red} = \concept{\_, round, \_}). The \emph{difference} between two concepts $C_1$ and $C_2$, where $C_1 \subset C_2$ is the concept $C_2 \setminus C_1$ (e.g. \concept{small, round, \_} \operator{IGNORE} \concept{\_, round, \_} = \concept{small, \_, \_}). These operators allow for a traversal over a broader set of concepts within the implicit hierarchy given knowledge of a limited training subset of concepts.

\section{Model architecture}
\paragraph{Learning visual representational primitives}
\label{sec_vae_dae}
The discovery of the generative structure of the visual world is the goal of disentangled factor learning research \citep{Bengio_etal_2013}. In this work we build SCAN\xspace on top of \ensuremath{\beta}-VAE\xspace, a state of the art model for unsupervised visual disentangled factor learning. \ensuremath{\beta}-VAE\xspace is a modification of the variational autoencoder (VAE) framework \citep{Kingma_Welling_2014, Rezende_etal_2014} that introduces an adjustable hyperparameter $\beta$ to the original VAE objective:
\begin{align} \label{eq_beta_vae}
    \mathcal{L}_x(\theta, \phi; \myvec{x}, \myvec{z}_x, \beta)  =  \mathbb{E}_{q_{\phi}(\myvec{z}_x|\myvec{x})} [\log p_{\theta}(\myvec{x} | \myvec{z}_x)] - \beta \ D_{KL}\divx{q_{\phi}(\myvec{z}_x|\myvec{x})}{p(\myvec{z}_x)}
\end{align}

where $\phi$, $\theta$ parametrise the distributions of the encoder and the decoder respectively. Well chosen values of $\beta$ (usually $\beta>1$) result in more disentangled latent representations $\myvec{z}_x$ by setting the right balance between reconstruction accuracy, latent channel capacity and independence constraints to encourage disentangling. For some datasets, however, this balance is tipped too far away from reconstruction accuracy. In these scenarios, disentangled latent representations $\myvec{z}_x$ may be learnt at the cost of losing crucial information about the scene, particularly if that information takes up a small proportion of the observations $\myvec{x}$ in pixel space. Hence, we adopt the solution used in \citet{Higgins_etal_2017b} that replaces the pixel log-likelihood term in Eq.~\ref{eq_beta_vae} with an L2 loss in the high-level feature space of a denoising autoencoder (DAE) \citep{Vincent_etal_2010} trained on the same data (see Fig.~\ref{fig_gsd_vae}C for model architecture). The resulting \ensuremath{\beta\text{-VAE}_{DAE}}\xspace architecture optimises the following objective function:
\begin{align} \label{eq_beta_vae_dae}
    \mathcal{L}_x(\theta, \phi; \myvec{x}, \myvec{z}_x, \beta)  = - \mathbb{E}_{q_{\phi}(\myvec{z}_x|\myvec{x})}\left \|J(\hat{\myvec{x}}) - J(\myvec{x})  \right \|_2^2 - \beta \ D_{KL}\divx{q_{\phi}(\myvec{z}_x|\myvec{x})}{p(\myvec{z}_x)}
\end{align}

where $\hat{\myvec{x}} \sim  p_{\theta}(\myvec{x} | \myvec{z}_x)$ and $J : \mathbb{R}^{W \times H \times C} \rightarrow  \mathbb{R}^N$ is the function that maps images from pixel space with dimensionality Width $\times$ Height $\times$ Channels to a high-level feature space with dimensionality $N$ given by a stack of DAE layers up to a certain layer depth (a hyperparameter). Note that this adjustment means that we are no longer optimising the variational lower bound, and \ensuremath{\beta\text{-VAE}_{DAE}}\xspace with $\beta=1$ loses its equivalence to the original VAE framework.

\paragraph{Learning visual concepts}
\label{sec_gsd_vae}
This section describes how our proposed SCAN\xspace framework (Fig.~\ref{fig_gsd_vae}A) exploits the particular parametrisation of the visual building blocks acquired by \ensuremath{\beta}-VAE\xspace\footnote{For the rest of the paper we use the term \ensuremath{\beta}-VAE\xspace to refer to \ensuremath{\beta\text{-VAE}_{DAE}}\xspace.} to learn an implicit hierarchy of visual concepts formalised in Sec.~\ref{sec_formalising}. SCAN\xspace is based on a modified VAE framework. In order to encourage the model to learn visually grounded abstractions, we initialise the space of concepts (the latent space $\myvec{z}_y$ of SCAN\xspace) to be structurally identical to the space of visual primitives (the latent space $\myvec{z}_x$ of \ensuremath{\beta}-VAE\xspace). Both spaces are parametrised as multivariate Gaussian distributions with diagonal covariance matrices, and $\text{dim}(\myvec{z}_y) = \text{dim}(\myvec{z}_x) = K$. The grounding is performed by aiming to minimise the KL divergence between the two distributions.

The abstraction step corresponds to setting SCAN\xspace latents $z_y^k$ corresponding to the relevant factors to narrow distributions, while defaulting those corresponding to the irrelevant factors to the wider unit Gaussian prior. This is done by minimising the forward KL divergence $D_{KL}\divx{q(\myvec{z}_x)}{q(\myvec{z}_y)}$, rather than the mode picking reverse KL divergence $D_{KL}\divx{q(\myvec{z}_y)}{q(\myvec{z}_x)}$. Fig.~\ref{fig_gsd_vae}B demonstrates the differences. Each blue mode corresponds to an inferred visual latent distribution $q(z_x^k | x_i)$ given an image $x_i$. The yellow distribution corresponds to the learnt conceptual latent distribution $q(z_y^k)$. When presented with visual examples that have high variability for a particular generative factor, e.g. various lighting conditions when viewing examples of apples, the forward KL allows SCAN\xspace to learn a broad distribution for the corresponding conceptual latent $q(z_y^k)$ that is close to the prior $p(z_y^k)=\mathcal{N}(0, 1)$. Hence, SCAN\xspace is trained by minimising:
\begin{align} \label{eq_gsd_vae}
    \mathcal{L}_y(\theta_y, \phi_y; \myvec{y}, \myvec{x}, \myvec{z}_y, \beta, \lambda) =&  \mathbb{E}_{q_{\phi_y}(\myvec{z}_y|\myvec{y})} [\log p_{\theta_y}(\myvec{y} | \myvec{z}_y)] \nonumber  - \beta \ D_{KL}\divx{q_{\phi_y}(\myvec{z}_y|\myvec{y})}{p(\myvec{z}_y)} \\
 & - \lambda \ D_{KL}\divx{q_{\phi_x}(\myvec{z}_x|\myvec{x})}{q_{\phi_y}(\myvec{z}_y|\myvec{y})}
\end{align}

\noindent where $\myvec{y}$ is symbol inputs, $\myvec{z}_y$ is the latent space of concepts, $\myvec{z}_x$ is the latent space of the pre-trained \ensuremath{\beta}-VAE\xspace containing the visual primitives which ground the abstract concepts $\myvec{z}_y$, and $\myvec{x}$ are example images that correspond to the concepts $\myvec{z}_y$ activated by symbols $\myvec{y}$. It is important to up-weight the forward KL term relative to the other terms in the cost function (e.g. $\lambda=1$, $\beta=10$).

The SCAN\xspace architecture does not make any assumptions on the nature of the symbols $\myvec{y}$. In this paper we use a commonly used k-hot encoding \citep{Vedantam_etal_2017, Suzuki_etal_2017}, where each concept is described in terms of the $k \leq K$ visual attributes it refers to (e.g. an apple could be referred to by a 3-hot symbol \sym{round, small, red}). In principle, other possible encoding schemes for $\myvec{y}$ can also be used, including word embeddings \citep{Mikolov_etal_2013}, or even entirely random vectors. We leave the empirical demonstration of this to future work.

Once trained, SCAN\xspace allows for bi-directional inference and generation (img2sym and sym2img). In order to generate visual samples that correspond to a particular concept (sym2img), we infer the concept $\myvec{z}_y$ by presenting an appropriate symbol $\myvec{y}$ to the inference network of SCAN\xspace. One can then sample from the inferred concept $q_{\phi_y}(\myvec{z}_y|\myvec{y})$ and use the generative part of \ensuremath{\beta}-VAE\xspace to visualise the corresponding image samples $p_{\theta_x}(\myvec{x} | \myvec{z}_y)$. SCAN\xspace can also be used to infer a description of an image in terms of the different learnt concepts via their respective symbols. To do so, an image $\myvec{x}$ is presented to the inference network of the \ensuremath{\beta}-VAE\xspace to obtain its description in terms of the visual primitives $\myvec{z}_x$. One then uses the generative part of the SCAN\xspace to sample descriptions $p_{\theta_y}(\myvec{y} | \myvec{z}_x)$ in terms of symbols that correspond to the previously inferred visual building blocks $q_{\phi_x}(\myvec{z}_x|\myvec{x})$.

\paragraph{Learning concept recombination operators}
\label{sec_recombinations}
The compositional and hierarchical structure of the concept latent space $\myvec{z}_y$ learnt by SCAN\xspace can be exploited to break away from the training data distribution and imagine new concepts. This can be done by using logical concept manipulation operators \operator{AND}, \operator{IN COMMON} and \operator{IGNORE} formally defined in Sec.~\ref{sec_formalising}. These operators are implemented within a conditional convolutional module parametrised by $\psi$ (Fig.~\ref{fig_imagining_recombinations}A) that accepts two multivariate Gaussian distributions $\myvec{z}_{y_1}$ and $\myvec{z}_{y_2}$ corresponding to the two concepts that are to be recombined, and a conditioning vector $\myvec{r}$ specifying the recombination operator. The input distributions $\myvec{z}_{y_1}$ and $\myvec{z}_{y_2}$ are inferred from the two corresponding input symbols $\myvec{y}_1$ and $\myvec{y}_2$, respectively, using a pre-trained SCAN\xspace. The convolutional module strides over the parameters of each matching component $z_{y_1}^k$ and $z_{y_2}^k$ one at a time and outputs the corresponding parametrised component $z_r^k$ of a recombined multivariate Gaussian distribution $\myvec{z}_r$ with a diagonal covariance matrix.\footnote{We also tried a closed form implementation of recombination operators (weighted sum or mean of the corresponding Gaussian components $z_{y_1}^k$ and $z_{y_2}^k$). We found that the learnt recombination operators worked better, achieving 0.79 vs 0.54 accuracy (higher is better) and 1.05 vs 2.03 diversity (lower is better) scores compared to the closed form implementations. See Sec.~\ref{sec_quant} for the description of the accuracy and diversity metrics.} We used 1-hot encoding for the conditioning vector $\myvec{r}$, where $\rowvec{1&0&0}$, $\rowvec{0&1&0}$ and $\rowvec{0&0&1}$ stood for \operator{AND}, \operator{IN COMMON} and \operator{IGNORE} respectively. The conditioning was implemented as a tensor product that takes in $\myvec{z}_{y_1}$ and $\myvec{z}_{y_2}$ and outputs $\myvec{z}_r$, where $\myvec{r}$ effectively selects the appropriate trainable transformation matrix parametrised by $\psi$. The conditional convolutional module is trained through the same visual grounding process as SCAN\xspace -- each recombination instruction is paired with a small number of appropriate example images (e.g. \sym{blue,suitcase} \operator{IGNORE} \sym{suitcase} might be paired with various example images containing a blue object). The recombination module is trained by minimising:
\begin{align} \label{eq_rec_mlp_loss}
\mathcal{L}_r(\psi; \myvec{z}_x, \myvec{z}_r)  =
    D_{KL}\big\lbrack q_{\phi_x}(\myvec{z}_x|\myvec{x}_i) ~\big|\big|~ q_{\psi}\left(\myvec{z}_r ~|~ q_{\phi_y}(\myvec{z}_{y_1}|\myvec{y}_1), q_{\phi_y}(\myvec{z}_{y_2}|\myvec{y}_2), \myvec{r}\right) \big\rbrack
\end{align}

\noindent where $q_{\phi_x}(\myvec{z}_x|\myvec{x}_i)$ is the inferred latent distribution of the \ensuremath{\beta}-VAE\xspace given a seed image $\myvec{x}_i$ that matches the specified symbolic description. The resulting $\myvec{z}_r$ lives in the same space as $\myvec{z}_y$ and corresponds to a node within the implicit hierarchy of visual concepts. Hence, all the properties of concepts $\myvec{z}_y$ discussed in the previous section also hold for $\myvec{z}_r$.

\begin{figure}[t]
 \centering
 \includegraphics[width = 1\textwidth]{figures/recombination_model_results_v6.pdf}
 \vspace{-5pt}
 \caption{\textbf{A}: Learning \operator{AND}, \operator{IN COMMON} or \operator{IGNORE} recombination operators with a SCAN\xspace model architecture. Inset demonstrates the convolutional recombination operator that takes in $\{\mu_{y_1}^k, \sigma_{y_1}^k; \mu_{y_2}^k, \sigma_{y_2}^k \}$ and outputs $\{\mu_r^k, \sigma_r^k \}$. The capital letters correspond to four disentangled visual primitives: object identity ($I$), object colour ($O$), floor colour ($F$) and wall colour ($W$). \textbf{B}: Visual samples produced by SCAN\xspace and JMVAE when instructed with a novel concept recombination. SCAN\xspace samples consistently match the expected ground truth recombined concept, while maintaining high variability in the irrelevant visual primitives. JMVAE samples lack accuracy. Recombination instructions are used to imagine concepts that have never been seen during model training. \textbf{Top}: samples for \operator{IGNORE}; \textbf{Middle}: samples for \operator{IN COMMON}; \textbf{Bottom}: samples for \operator{AND}.}
 \label{fig_imagining_recombinations}
\end{figure}

\section{Introduction}
\label{sec_intro}
State of the art deep learning approaches to machine learning have achieved impressive results in many problem domains, including classification \citep{He_etal_2016, Szegedy_etal_2015}, density modelling \citep{Gregor_etal_2015, Oord_etal_2016, Oord_etal_2016b}, and reinforcement learning \citep{Mnih_etal_2015, Mnih_etal_2016, Jaderberg_etal_2017, Silver_etal_2016}. They are still, however, far from possessing many traits characteristic of human intelligence. Such deep learning techniques tend to be overly data hungry, often rely on significant human supervision and tend to overfit to the training data distribution \citep{Lake_etal_2016, Garnelo_etal_2016}. An important step towards bridging the gap between human and artificial intelligence is endowing algorithms with compositional concepts \citep{Lake_etal_2016, Garnelo_etal_2016}. Compositionality allows for reuse of a finite set of primitives (addressing the data efficiency and human supervision issues) across many scenarios by recombining them to produce an exponentially large number of novel yet coherent and potentially useful concepts (addressing the overfitting problem). Compositionality is at the core of such human abilities as creativity, imagination and language-based communication.

\begin{figure}[t]
 \centering
 \includegraphics[width = 0.9\textwidth]{figures/concept_hierarchy_dmlab_v4.pdf}
 \vspace{-6pt}
 \caption{Schematic of an implicit concept hierarchy built upon a subset of four visual primitives: object identity ($I$), object colour ($O$), floor colour ($F$) and wall colour ($W$) (other visual primitives necessary to generate the scene are ignored in this example). Concepts form an implicit hierarchy, where each parent is an abstraction over its children and over the original set of visual primitives (the values of the concept-defining sets of visual primitives are indicated by the bold capital letters). In order to generate an image that corresponds to a concept, one has to fill in values for the factors that got abstracted away (represented as \enquote{\_}), e.g. by sampling from their respective priors. Given certain nodes in the concept hierarchy, one can traverse the other nodes using logical operations. See Sec.\ref{sec_formalising} for our formal definition of concepts.}
 \label{fig_concept_hierarchy}
\end{figure}

We propose that concepts are abstractions over a set of primitives. For example, consider a toy hierarchy of visual concepts shown in Fig.~\ref{fig_concept_hierarchy}. Each node in this hierarchy is defined as a subset of visual primitives that make up the scene in the input image. These visual primitives might include factors like object identity, object colour, floor colour and wall colour. As one traverses the hierarchy from the subordinate over basic to superordinate levels of abstraction \citep{Rosch_1978} (i.e. from the more specific to the more general concepts corresponding to the same visual scene), the number of concept-defining visual primitives decreases. Hence, each parent concept in such a hierarchy is an abstraction (i.e. a subset) over its children and over the original set of visual primitives. A more formal definition of concepts is provided in Sec.~\ref{sec_formalising}.

Intelligent agents are able to discover and learn abstract compositional concepts using little supervision \citep{Baillargeon_1987, Spelke_1990, Baillargeon_2004, Smith_Vul_2013}. Think of human word learning -- we acquire the meaning of words through a combination of a continual stream of unsupervised visual data occasionally paired with a corresponding word label. This paper describes SCAN\xspace (Symbol-Concept Association Network, see Fig.~\ref{fig_gsd_vae}A), a neural network model capable of learning grounded visual concepts in a largely unsupervised manner through fast symbol association. First, we use the \ensuremath{\beta}-VAE\xspace \citep{Higgins_etal_2017} to learn a set of independent representational primitives through unsupervised exposure to the visual data. This is equivalent to learning a disentangled (factorised and interpretable) representation of the independent ground truth \enquote{generative factors} of the data \citep{Bengio_etal_2013}. Next, we allow SCAN\xspace to discover meaningful abstractions over these disentangled primitives by exposing it to a small number of symbol-image pairs that apply to a particular concept (e.g. a few example images of an apple paired with the symbol \sym{apple}). SCAN\xspace learns the meaning of the concept by identifying the set of visual primitives that all the visual examples have in common (e.g. all observed apples are small, round and red). The corresponding symbol (\sym{apple}) then becomes a \enquote{pointer} to the newly acquired concept \concept{small, round, red} - a way to access and manipulate the concept without having to know its exact representational form. Our approach does not make any assumptions about how these symbols are encoded, which also allows SCAN\xspace to learn multiple referents to the same concept, i.e. synonyms.

Once a concept is acquired, it should be possible to use it for bi-directional inference: the model should be able to generate diverse visual samples that correspond to a particular concept (\emph{sym2img}) and vice versa (\emph{img2sym}). Since the projection from the space of visual primitives to the space of concepts (img2sym, red dash arrow in Fig.~\ref{fig_concept_hierarchy}) involves abstraction and hence a loss of information, one then needs to add compatible information back in when moving from the space of concepts to that of visual primitives (sym2img, blue dot arrow in Fig.~\ref{fig_concept_hierarchy}). In our setup, concepts are defined in terms of a set of relevant visual primitives (e.g. colour, shape and size for \sym{apple}). This leaves a set of irrelevant visual attributes (e.g. lighting, position, background) to be \enquote{filled in}. We do so by defaulting them to their respective priors, which ensures high diversity of samples (in both image or symbol space) for each concept during img2sym and sym2img inferences.

The structured nature of learnt concepts acquired by SCAN\xspace allows for sample efficient learning of logical recombination operators: \operator{AND} (corresponding to a set union of relevant primitives), \operator{IN COMMON} (corresponding to set intersection) and \operator{IGNORE} (corresponding to set difference), by pairing a small number of valid visual examples of recombined concepts with the respective operator names. Once the meaning of the operators has been successfully learned, SCAN\xspace can exploit the compositionality of the acquired concepts, and traverse previously unexplored parts of the implicit underlying concept hierarchy by manipulating and recombining existing concepts in novel ways. For example, a new node corresponding to the concept \concept{blue, small} can be reached through the following instructions: \sym{blue} \operator{AND} \sym{small} (going down the hierarchy from more general to more specific), \sym{blueberry} \operator{IN COMMON} \sym{bluebell} (going up the hierarchy from more specific to more general) or \sym{blueberry} \operator{IGNORE} \sym{round} (also going up the hierarchy).

To summarise, our paper 1) presents SCAN\xspace, a neural network model capable of learning compositional and hierarchical representations of visual concepts; 2) demonstrates that SCAN\xspace can be successfully trained with very little supervised data; 3) shows that after training, SCAN\xspace can perform multimodal (visual and symbolic) bi-directional inference and generation with high accuracy and diversity, outperforming all baselines; 4) shows that the addition of logical recombination operations allows SCAN\xspace to break out of its limited training data distribution and reach new nodes within the implicit hierarchy of concepts.

\begin{figure}[t]
 \centering
 \includegraphics[width = \textwidth]{figures/beta_vae_gsd_vae_kl_assymmetry_v2.pdf}
 \vspace{-6pt}
 \caption{\textbf{A}: SCAN\xspace model architecture. The capital letters correspond to four disentangled visual primitives: object identity ($I$), object colour ($O$), floor colour ($F$) and wall colour ($W$). \textbf{B}: Mode coverage of the extra KL term of the SCAN\xspace loss function. Forward KL divergence $D_{KL}\divx{\myvec{z}_x}{\myvec{z}_y}$ allows SCAN\xspace to learn abstractions (wide yellow distribution $\myvec{z}_y$) over the visual primitives that are irrelevant to the meaning of a concept (blue modes corresponds to the inferred values of $\myvec{z}_x$ for different visual examples matching symbol $\myvec{y}$). \textbf{C}: \ensuremath{\beta\text{-VAE}_{DAE}}\xspace model architecture.}
 \label{fig_gsd_vae}
\end{figure}

\section{Related work}
To the best of our knowledge no framework currently exists that is directly equivalent to SCAN\xspace. Past relevant literature can broadly be split into three categories: 1) Bayesian models that try to mimic fast human concept learning \citep{Tennenbaum_1999, Lake_etal_2015}; 2) conditional generative models that aim to generate faithful images conditioned on a list of attributes or other labels \citep{Reed_etal_2016, Reed_etal_2016b, Kingma_etal_2014, Yan_etal_2016, Sohn_etal_2015, Pandey_Dukkipati_2017} ; and 3) multimodal generative models that aim to embed visual and symbolic inputs in a joint latent space in order to be able to run bi-directional inferences \citep{Vedantam_etal_2017, Suzuki_etal_2017, Pu_etal_2016, Wang_etal_2016, Srivastava_Salakhutdinov_2014}. Bayesian models by \citet{Tennenbaum_1999} and \citet{Lake_etal_2015} can learn from few examples, but, unlike SCAN\xspace, they are not fully grounded in visual data. Conditional and joint multimodal models are fully grounded in visual data, however, unlike SCAN\xspace, they require a large number of image-symbol pairs for training. An exception to this is the model by \cite{Srivastava_Salakhutdinov_2014}, which, however, cannot generate images, instead relying on feature-guided nearest-neighbour lookup within existing data, and also requires slow MCMC sampling. Multimodal generative models are capable of bi-directional inference, however they tend to learn a flat unstructured latent space unlike the hierarchical compositional latent space of SCAN\xspace. Hence these baselines underperform SCAN\xspace in terms of sample diversity and the ability to break out of their training data distribution through symbolically instructed logical operations.

\section{Supplementary Information}
\subsection{Model details}

\paragraph{\ensuremath{\beta}-VAE\xspace} We re-used the architecture and the training setup for \ensuremath{\beta}-VAE\xspace specified in \cite{Higgins_etal_2017b}. In particular, we used L2 loss within a pre-trained denoising autoencoder (DAE) to calculate the reconstruction part of the \ensuremath{\beta}-VAE\xspace loss function. The DAE was trained with occlusion-style masking noise in the vein of \citet{Pathak_etal_2016}. Concretely, two values were independently sampled from $U[0, W]$ and two from $U[0, H]$ where $W$ and $H$ were the width and height of the input frames. These four values determined the corners of the rectangular mask applied; all pixels that fell within the mask were set to zero.

The DAE architecture consisted of four convolutional layers, each with kernel size $4$ and stride $2$ in both the height and width dimensions. The number of filters learnt for each layer was $\{32, 32, 64, 64\}$ respectively. The bottleneck layer consisted of a fully connected layer of size $100$ neurons. This was followed by four deconvolutional layers, again with kernel sizes $4$, strides $2$, and $\{64, 64, 32, 32\}$ filters. The padding algorithm used was `SAME' in TensorFlow \citep{Abadi_etal_2015}. ELU non-linearities were used throughout. The optimiser used was Adam \citep{Kingma_Ba_2014} with a learning rate of $\num{1e-3}$ and $\epsilon=\num{1e-8}$. We pre-trained the DAE for $200,000$ steps, using batch size of $100$ before training \ensuremath{\beta}-VAE\xspace.

\ensuremath{\beta}-VAE\xspace architecture was the following. We used an encoder of four convolutional layers, each with kernel size $4$, and stride $2$ in the height and width dimensions. The number of filters learnt for each layer was $\{32, 32, 64, 64\}$ respectively. This was followed by a fully connected layer of size $256$ neurons. The latent layer comprised $64$ neurons parametrising $32$ (marginally) independent Gaussian distributions. The decoder architecture was simply the reverse of the encoder, utilising deconvolutional layers. The decoder used was Bernoulli. The padding algorithm used was `SAME' in TensorFlow. ReLU non-linearities were used throughout. The reconstruction error was taking in the last layer of the DAE (in the pixel space of DAE reconstructions) using L2 loss and before the non-linearity. The optimiser used was Adam with a learning rate of $\num{1e-4}$ and $\epsilon=\num{1e-8}$. We pre-trained \ensuremath{\beta}-VAE\xspace until convergence using batch size of $100$. The disentangled \ensuremath{\beta}-VAE\xspace had $\beta=53$, while the entangled \ensuremath{\beta}-VAE\xspace used within the \ensuremath{\text{SCAN}_\text{U}}\xspace baseline had $\beta=0.1$.

\paragraph{SCAN\xspace}
The encoder and decoder of SCAN\xspace were simple single layer MLPs with $100$ hidden units for DeepMind Lab experiments and a two layer MLP with $500$ hidden units in each hidden layer for the CelebA experiments. We used ReLU non-linearities in both cases. The decoder was parametrised as a Bernoulli distribution over the output space of size $375$. We set $\beta_y=1$ for all experiments, and  $\lambda=10$. We trained the model using Adam optimiser with learning rate of \num{1e-4} and batch size $16$.

\paragraph{SCAN\xspace recombination operator}
The recombination operator was implemented as a convolutional operator with kernel size $1$ and stride $1$. The operator was parametrised as a $2$ layer MLP with $30$ and $15$ hidden units per layer, and ReLU non-linearities. The optimizer is Adam with a learning rate of \num{1e-3} and batch size $16$ was used. We trained the recombination operator for $50k$ steps.

\paragraph{JMVAE}
The JMVAE was trained using the loss as described in \citep{Suzuki_etal_2017}:

\begin{align}
    \mathcal{L}_{JM}(\theta_x, \theta_y, \phi_x, \phi_y, \phi; \myvec{x}, \myvec{y}, \alpha) = &  \mathbb{E}_{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})} \left[\log p_{\theta_x}(\myvec{x} | \myvec{z}) \right] ~+~ \mathbb{E}_{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})} \left[\log p_{\theta_y}(\myvec{y} | \myvec{z}) \right]  \nonumber \\
    & - D_{KL}\divx{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})}{p(\myvec{z})} \nonumber \\
    & - \alpha \big\lbrack D_{KL}\divx{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})}{q_{\phi_x}(\myvec{z}|\myvec{x})} \nonumber \\
    &\qquad + D_{KL}\divx{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})}{q_{\phi_y}(\myvec{z}|\myvec{y})} \big\rbrack \label{eq_jmvae}
\end{align}

Where $\alpha$ was a hyperparameter. We tried $\alpha$ values $\{0.01, 0.1, 1.0, 10.0\}$ as in the original paper and found that the best results were obtained with $\alpha = 1.0$. All results were reported with this value.

The architectural choices for JMVAE were made to match as closely as possible those made for SCAN\xspace. Thus the visual encoder $q_{\phi_x}$ consisted of four convolutional layers, each with kernel size $4$ and stride $2$ in both the height and width dimensions, with $\{32, 32, 64, 64\}$ filters learned at the respective layers. The convolutional stack was followed by a single fully connected layer with $256$ hidden units. The encoder output the parametrisation for a a 32-dimensional diagonal Gaussian latent distribution. The symbol encoder $q_{\phi_y}$ consisted of a single layer MLP with $100$ hidden units for the DeepMind Lab experiments or two layer MLP with $500$ hidden units per layer for the CelebA experiments as in SCAN\xspace. The joint encoder $q_{\phi}$ consisted of the same convolutional stack as in the visual encoder to process the visual input, while the symbol input was passed through a two-layer MLP with $32$ and $100$ hidden units. These two embeddings were then concatenated and passed through a further two-layer MLP of $256$ hidden units each, before outputting the $64$ parameters of the diagonal Gaussian latents.

The visual decoder $p_{\theta_x}$ was simply the reverse of the visual encoder using transposed convolutions. Similarly, the symbol decoder $p_{\theta_x}$ was again a single layer MLP with $100$ hidden units. The output distributions of both decoders were parameterised as Bernoullis. The model was trained using the Adam optimiser with a learning rate of \num{1e-4} and a batch size of $16$.

\paragraph{Triple ELBO (TrELBO)}
The Triple ELBO (TrELBO) model was trained using the loss as described in \citep{Vedantam_etal_2017}:

\begin{align}
    \mathcal{L}_{trelbo}(\theta_x, \theta_y, \phi_x, \phi_y, \phi; \myvec{x}, \myvec{y}, \lambda_y^{yx}, \lambda_y^y) =&~ \mathbb{E}_{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})} \left[\log p_{\theta_x}(\myvec{x} | \myvec{z}) \right] ~+~ \mathbb{E}_{q_{\phi_x}(\myvec{z}|\myvec{x})} \left[\log p_{\theta_x}(\myvec{x} | \myvec{z}) \right]  \nonumber \\
    & + \lambda_y^{yx} \mathbb{E}_{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})} \left[\log p_{\theta_y}(\myvec{y} | \myvec{z}) \right] + \lambda_y^y \mathbb{E}_{q_{\phi_y}(\myvec{z}|\myvec{y})} \left[\log p_{\theta_y}(\myvec{y} | \myvec{z}) \right]  \nonumber \\
    & - D_{KL}\divx{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})}{p(\myvec{z})} - D_{KL}\divx{q_{\phi_x}(\myvec{z}|\myvec{x})}{p(\myvec{z})} \nonumber \\
    & - D_{KL}\divx{q_{\phi_y}(\myvec{z}|\myvec{y})}{p(\myvec{z})}
    \label{eq_trelbo}
\end{align}

Where $\lambda_y^{yx}$ and $\lambda_y^{y}$ were hyperparameters. We set these to 10 and 100 respectively, following the reported best values from \citep{Vedantam_etal_2017}.

We trained the model using the frozen-likelihood trick shown to improve the model performance in \citet{Vedantam_etal_2017}. The symbol decoder parameters $\theta_y$ were trained only using the $\mathbb{E}_{q_{\phi}(\myvec{z}|\myvec{x}, \myvec{y})} \left[\log p_{\theta_y}(\myvec{y} | \myvec{z})\right]$ term and not the $\mathbb{E}_{q_{\phi_y}(\myvec{z}|\myvec{y})} \left[\log p_{\theta_y}(\myvec{y} | \myvec{z}) \right]$ term. For fair comparison to the other models, we did not utilise a product of experts for the inference network $q_{\phi_y}(\myvec{z}|\myvec{y})$.

In all architectural respects, the networks used were identical to those reported above for the JMVAE. The same training procedures were followed.

\paragraph{Accuracy and diversity evaluation}
\label{sup_acc_div_details}
The classifier used to evaluate the samples generated by each model was trained to discriminate the four room configuration factors in the DeepMind Lab dataset: wall colour, floor colour, object colour and object identity.  We used a network of four 2-strided deconvolutional layers (with filters in each successive layer of $\{32, 64, 128, 256\}$, and kernels sized 3x3), followed by a fully connected layer with $256$ neurons, with ReLU activations used throughout. The output layer consisted of four fully connected softmax heads, one for each predicted factor (with dimensionality $16$ for each of the colour factors, $3$ for object identity). The classifier was trained until convergence using the Adam optimiser, with a learning rate of \num{1e-4} and batch size of $100$ (reaching an overall accuracy of $0.992$).

The accuracy metric for the sym2img samples was computed as the average top-k accuracy across the factors (with $k=3$ for the colour factors, and $k=1$ for the object identity factor), against the ground-truth factors specified by the concept used to generate each sym2img sample. The top-k of the factors in each image sample was calculated using the top-k softmax outputs of the classifier.

Sample diversity of the sym2img data was characterised by estimating the KL divergence of the irrelevant factor distribution inferred for each concept with a flat distribution, $D_{KL}\divx{u(\myvec{y}_i)}{p(\myvec{y}_i)}$. Here, $p(\myvec{y}_i)$ is the joint distribution of the irrelevant factors in the sym2img set of images generated from the $i$th concept, which we estimated by averaging the classifier predictions across those images. $u(\myvec{y}_i)$ is the desired (flat) joint distribution of the same factors (i.e., where each factor value has equal probability). We also computed the expected KL if $p(\myvec{y}_i)$ were estimated using the samples drawn from the flat distribution $u(\myvec{y}_i)$. We report the mean of this KL across all the k-grams. We used $64$ sym2img samples per concept.

\subsection{DeepMind Lab Dataset details}

\paragraph{RGB to HSV conversion}
\label{sup_rgb_hsv}
The majority of the data generative factors to be learnt in the DeepMind Lab dataset correspond to colours (floor, wall and object). We found that it was hard to learn disentangled representations of these data generative factors with \ensuremath{\beta}-VAE\xspace. We believe this is because \ensuremath{\beta}-VAE\xspace requires a degree of smoothness in pixel space when traversing a manifold for a particular data generative factor in order to correctly learn this factor \citep{Higgins_etal_2017}. The intuitively smooth notion of colour, however, is disrupted in RGB space (see Fig.~\ref{fig_rgb_hsv}). Instead, the intuitive human notion of colour is more closely aligned with hue in HSV space. Hence we added a pre-processing step that converted the DeepMind Lab frames from the RGB to HSV space before training \ensuremath{\beta}-VAE\xspace. This conversion preserved the dimensionality of the frames, since both RGB and HSV require three channels. We found that this conversion enabled \ensuremath{\beta}-VAE\xspace to achieve good disentangling results.

\begin{figure}[th]
 \centering
 \includegraphics[width = 1\textwidth]{figures/rgb_hsv_v2.pdf}
 \vspace{-6pt}
 \caption{\textbf{A}: Comparison of hue traversal in HSV space, which closely aligns with the intuitive human understanding of colour, and the equivalent highly non-monotonic changes in RGB space. \emph{H} stands for hue, \emph{S} stands for saturation and \emph{V} stands for value/brightness. Adapted from \cite{hsv_wiki}. \textbf{B}: Visualisation of colours used in DeepMind Lab in RGB. \textbf{C}: Visualisation of colours used in DeepMind Lab in HSV. It can be seen that the HSV projection of the DeepMind Lab colours appears significantly more structured than the equivalent RGB projection.}
 \label{fig_rgb_hsv}
\end{figure}

\paragraph{k-hot experiments}
Our DeepMind Lab \citep{deepmind_lab} dataset contained $73$ frames per room, where the configuration of each room was randomly sampled from the outer product of the four data generative factors: object identity and colour, wall and floor colour ($18,883$ unique factor combinations). All models were trained using a randomly sampled subset of $133$ concepts (with $10$ example images per concept), $30$ extra concepts were used for training the recombination operators ($20$ example images per concept) and a further set of $50$ concepts were used to evaluate the models' ability to break away from their training distribution using recombination operators.

\paragraph{Training the recombination operator}
\label{sup_recomb_details}
The recombination operator was trained by sampling two concepts, $\myvec{y}_1$ and $\myvec{y}_2$, and an operator $\myvec{r}$ as input. The training objective was to ground $\myvec{z}_r$ in the ground truth latent space $\myvec{z}_x$ inferred from an image $\myvec{x}$. The ground truth image $\myvec{x}$ was obtained by applying binary logical operation corresponding to $\myvec{r}$ to binary symbols $\myvec{y}_1$ and $\myvec{y}_2$. This produces the ground truth recombined symbol $\myvec{y}_r$, which can then be used to fetch a corresponding ground truth image $\myvec{x}_r$ from the dataset.

To make sure that the logical operators were not presented with nonsensical instructions, we followed the following logic for sampling minibatches of $\myvec{y}_1$ and $\myvec{y}_2$ during training. The \operator{IN COMMON} and \operator{AND} operators were trained by sampling two k-grams $\myvec{y}_1$ and $\myvec{y}_2$ with $k \in \{1, 2, 3\}$. The \operator{IN COMMON} operator had an additional restriction that the intersection cannot be and empty set. The \operator{IGNORE} operator was trained by sampling a k-gram with $k \in \{1, 2, 3\}$ and a unigram selected from one of the factors specified by the k-gram.

\subsection{DeepMind Lab experiments}
\label{sup_baselines}
\paragraph{Unsupervised visual representation learning}
SCAN\xspace relies on the presence of structured visual primitives. Hence, we first investigate whether \ensuremath{\beta}-VAE\xspace trained in an unsupervised manner on the visually complex DeepMind Lab dataset has discovered a disentangled representation of all its data generative factors. As can be seen in Fig.~\ref{fig_dm_disentangling} (left panel), SCAN\xspace has learnt to represent each of the object-, wall-, and floor-colours, using two latents -- one for hue and one for brightness. Learning a disentangled representation of colour is challenging, but we were able to achieve it by projecting the input images from RGB to HSV space, which is better aligned with human intuitions of colour (see Sec.~\ref{sup_rgb_hsv}). We noticed that \ensuremath{\beta}-VAE\xspace confused certain colours (e.g. red floors are reconstructed as magenta, see the top right image in the Reconstructions pane of Fig.~\ref{fig_dm_disentangling}). We speculate that this is caused by trying to approximate the circular hue space using a linear latent. Red and magenta end up on the opposite ends of the linear latent while being neighbours in the circular space.

\begin{figure}[t]
 \centering
 \includegraphics[width = 1.0\textwidth]{figures/deepmind_lab_beta_vae_traversals.jpg}

 \vspace{-6pt}
 \caption{Reconstructions, samples and latent traversals of \ensuremath{\beta}-VAE\xspace ($\beta=53$) trained to disentangle the data generative factors of variation within the DeepMind Lab dataset. For the latent traversal plots we sampled the posterior, then visualised \ensuremath{\beta}-VAE\xspace reconstructions while resampling each latent unit one at a time in the $[-3, 3]$ range while keeping all other latents fixed to their originally sampled values. This process helps visualise which data generative factor each latent unit has learnt to represent. }
 \label{fig_dm_disentangling}
\end{figure}

 Compare the disentangled representations of Fig.\ \ref{fig_dm_disentangling} to the entangled equivalents in Fig.~\ref{fig_deepmind_lab_beta_vae_ent_traversals_ent}. Fig.~\ref{fig_deepmind_lab_beta_vae_ent_traversals_ent} shows that an entangled \ensuremath{\beta}-VAE\xspace was able to reconstruct the data well, however due to the entangled nature of its learnt representations, latent traversal plots and samples are not as good as those of a disentangled \ensuremath{\beta}-VAE\xspace (Fig.~\ref{fig_dm_disentangling}).

\begin{figure}[th]
 \centering
 \includegraphics[width = 1.0\textwidth]{figures/deepmind_lab_beta_vae_traversals_ent.jpg}
 \vspace{-6pt}
 \caption{Samples, reconstructions and latent traversals of \ensuremath{\beta}-VAE\xspace that did not learn a structured disentangled representation ($\beta=0.1$). It is evident that the model learnt to reconstruct the data despite learning an entangled latent space.}
 \label{fig_deepmind_lab_beta_vae_ent_traversals_ent}
\end{figure}

\paragraph{\ensuremath{\text{SCAN}_\text{U}}\xspace analysis}
As shown in Fig.~\ref{fig_deepmind_lab_beta_vae_ent_traversals_ent} SCAN\xspace with unstructured vision is based on a \ensuremath{\beta}-VAE\xspace that learnt a good (yet entangled) representation of the DeepMind Lab dataset. Due to the unstructured entangled nature of the visual latent space $\myvec{z}_x$, the additional forward KL term of the SCAN\xspace loss function (Eq.~\ref{eq_gsd_vae}) is not able to pick out the relevant visual primitives for each training concept. Instead, all latents end up in the irrelevant set, since the relevant and irrelevant ground truth factors end up being entangled in the latent space $\myvec{z}_x$. This disrupts the ability of SCAN\xspace with entangled vision to learn useful concepts, as demonstrated in Fig.~\ref{fig_dmlab_ent_sym2img}.

\begin{figure}[th]
 \centering
 \includegraphics[width = 0.8\textwidth]{figures/SCANu_samples_v2.png}
 \vspace{-6pt}
 \caption{Visual samples (sym2img) of SCAN\xspace grounded in unstructured vision when presented with symbols \sym{hat} and \sym{ice lolly}. It is evident that the model struggled to learn a good understanding of the meaning of these concepts.}
 \label{fig_dmlab_ent_sym2img}
\end{figure}

\paragraph{JMVAE analysis}
In this section we provide some insights into the nature of representations learnt by JMVAE \citep{Suzuki_etal_2017}. Fig.~\ref{fig_deepmind_lab_jmvae_traversals_ent} demonstrates that after training JMVAE is capable of reconstructing the data and drawing reasonable visual samples. Furthermore, the latent traversal plots indicate that the model learnt a reasonably disentangled representation of the data generative factors. Apart from failing to learn a latent to represent the spawn animation and a latent to represent all object identities (while the hat and the ice lolly are represented, the suitcase is missing), the representations learnt by JMVAE match those learnt by \ensuremath{\beta}-VAE\xspace (compare Figs.~\ref{fig_dm_disentangling} and \ref{fig_deepmind_lab_jmvae_traversals_ent}). Note, however, that unlike \ensuremath{\beta}-VAE\xspace that managed to discover and learn a disentangled representation of the data generative factors in a completely unsupervised manner, JMVAE was able to achieve its disentangling performance by exploiting the extra supervision signal coming from the symbolic inputs.

JMVAE is unable to learn a hierarchical compositional latent representation of concepts like SCAN\xspace does. Instead, it learns a flat representation of visual primitives like the representation learnt by \ensuremath{\beta}-VAE\xspace. Such a flat representation is problematic, as evidenced by the accuracy/diversity metrics shown in Tbl.~\ref{tbl_quant_results}. Further evidence comes from examining the sym2img samples produced by JMVAE (see Fig.~\ref{fig_dmlab_jmvae_sym2img}). It can be seen that JMVAE fails to learn the abstract concepts as defined in Sec.~\ref{sec_formalising}. While the samples in Fig.~\ref{fig_dmlab_jmvae_sym2img} mostly include correct wall colours that match their respective input symbols, the samples have limited diversity. Many samples are exact copies of each other -- a sign of mode collapse.

\begin{figure}[th]
 \centering
 \includegraphics[width = 1.0\textwidth]{figures/jmvae_recs.jpg}
 \vspace{-6pt}
 \caption{Samples, reconstructions and latent traversals of JMVAE. The model learns good disentangled latents, making use of the supervised symbolic information available.}
 \label{fig_deepmind_lab_jmvae_traversals_ent}
\end{figure}

\begin{figure}[th]
 \centering
 \includegraphics[width = 0.8\textwidth]{figures/jmvae_samples.png}
 \vspace{-6pt}
 \caption{Visualisation of sym2img visual samples produced JMVAE in response to symbols specifying wall colour names. It is evident that the model suffers from mode collapse, since a significant number of samples are copies of each other. }
 \label{fig_dmlab_jmvae_sym2img}
\end{figure}

\paragraph{TrELBO analysis}
This section examines the nature of representations learnt by TrELBO \citep{Vedantam_etal_2017}. Fig.~\ref{fig_deepmind_lab_trelbo_traversals_ent} demonstrates that after training TrELBO is capable of reconstructing the data, however it produces poor samples. This is due to the highly entangled nature of its learnt representation, as also evidenced by the traversal plots. Since TrELBO is not able to learn a compositional latent representation of concept like that acquired by SCAN\xspace, it also struggles to produce diverse sym2img samples when instructed with symbols from the training set (see Fig.~\ref{fig_dmlab_trelbo_sym2img}). Furthermore, this lack of structure in the learnt concept representations precludes successful recombination operator training. Hence, sym2img samples of test symbols instructed through recombination operators lack accuracy (Fig.~\ref{fig_dmlab_trelbo_sym2img_recomb}).

\begin{figure}[t!]
 \centering
 \includegraphics[width = 1.0\textwidth]{figures/deepmind_lab_trelbo_traversals.jpg}
 \vspace{-6pt}
 \caption{Samples, reconstructions and latent traversals of TrELBO. The model learns a very entangled representation.}
 \label{fig_deepmind_lab_trelbo_traversals_ent}
\end{figure}

\begin{figure}[t!]
 \centering
 \includegraphics[width = 0.8\textwidth]{figures/trelbo_samples.jpg}
 \vspace{-6pt}
 \caption{Visualisation of sym2img visual samples produced TrELBO in response to train symbols: \sym{magenta object}, \sym{ice lolly}, \sym{purple floor} and \sym{blue wall}. It is evident that the model has good accuracy but very low diversity. }
 \label{fig_dmlab_trelbo_sym2img}
\end{figure}

\begin{figure}[t!]
 \centering
 \includegraphics[width = 0.8\textwidth]{figures/trelbo_samples_recomb.jpg}
 \vspace{-6pt}
 \caption{Visualisation of sym2img visual samples produced TrELBO in response to test symbols instructed using recombination operators: \sym{yellow object}, \sym{hat}, \sym{orange floor} and \sym{cyan wall}. It is evident that the model has very low accuracy but decent diversity. }
 \label{fig_dmlab_trelbo_sym2img_recomb}
\end{figure}

\paragraph{Data efficiency analysis}
\label{sec_data_efficiency}
We evaluate the effect of the training set size on the performance of SCAN\xspace, JMVAE and TrELBO by comparing their accuracy and diversity scores after training on \{5, 10, 15, 20, 25, 50, 75\} concepts. Fig.~\ref{fig_sample_complexity} shows that SCAN\xspace consistently outperforms its baselines in terms of the absolute scores, while also displaying less variance when trained on datasets of various sizes. For this set of experiments we also halved the number of training iterations for all models, which affected the baselines but not SCAN\xspace. The diversity of JMVAE and TrELBO is better in this plot compared to the results reported in Tbl.~\ref{tbl_quant_results} because sym2img samples used for this plot were blurrier than those describe in the main text.

\begin{figure}[t!]
 \centering
 \includegraphics[width = 0.8\textwidth]{figures/Sample_complexity_comparison}
 \vspace{-6pt}
 \caption{Accuracy and diversity scores of SCAN\xspace, JMVAE and TrELBO after being trained on \{5, 10, 15, 20, 25, 50, 75\} concepts with 10 visual examples each. The size of the circle corresponds to the training set size. We used symbols from the train set to generate sym2img samples used to calculate the scores. SCAN\xspace outperforms both baselines and shows less susceptibility to the training set size.}
 \label{fig_sample_complexity}
\end{figure}

\subsection{dSprites experiments}
\label{sec_exp_dsprites}
In this section we describe additional experiments testing SCAN\xspace on the dSprites \citep{dsprites17} dataset. The dataset consists of binary sprites fully specified by five ground truth factors: position x (32 values), position y (32 values), scale (6 values), rotation (40 values) and sprite identity (3 values). For our experiments we defined a conceptual space spanned by three of the data generative factors - horizontal and vertical positions, and scale. We quantised the values of each chosen factor into halves (top/bottom, left/right, big/small) and assigned one-hot encoded symbols to each of the $\sum_{k=1}^K \binom{K}{k} N^k = 26$ possible concepts to be learnt (since $K=3$ is the number of factors to be learnt and $N=2$ is the number of values each factor can take). We compared the performance of SCAN\xspace grounded in disentangled visual representations (\ensuremath{\beta}-VAE\xspace with $\beta=12$) to that of \ensuremath{\text{SCAN}_{\textup{IS}}}\xspace grounded in entangled visual representations (\ensuremath{\beta}-VAE\xspace with $\beta=0$). We trained both models on a random subset of image-symbol pairs $(x_i, y_i)$ making up $<0.01\%$ of the full dataset.

We quantified how well the models understood the meaning of the positional and scale concepts after training by running sym2img inference and counting the number of white pixels within each of the four quadrants of the canvas (for position) or in total in the whole image (for scale). This can be compared to similar values calculated over a batch of ground truth images that match the same input symbols. Samples from SCAN\xspace matched closely the statistics of the ground truth samples (see Fig.~\ref{fig_dsprites_concept_understanding}). \ensuremath{\text{SCAN}_\text{U}}\xspace, however, failed to produce meaningful samples despite being able to reconstruct the dataset almost perfectly.

\begin{figure}[th]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/dSprites_sym2img.pdf}
 \vspace{-6pt}
 \caption{sym2img inference performance of SCAN\xspace and \ensuremath{\text{SCAN}_\text{U}}\xspace for symbols - ``left'' and ``large top''. First line in each subplot demonstrates ground truth samples from dSprites dataset that correspond to the respective symbol. Next three lines illustrate the comparative performance of SCAN\xspace (left) vs \ensuremath{\text{SCAN}_\text{U}}\xspace (right), including their respective sym2img samples, as well as the quantitative comparison of each model (green) to the ground truth (red) in terms of scale understanding (each bar corresponds to the average number of pixels per sample image) and positional understanding (each bar corresponds to the average number of pixels in one of the four quadrants of the samples: T - top, B - bottom, R - right, L - left). The closer the green bars are to the red bars, the better the model's understanding of the learnt concepts.}
 \label{fig_dsprites_concept_understanding}
\end{figure}

\subsection{CelebA experiments}

\begin{figure}[th]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/celeba_baselines_attributes_large.png}
 \vspace{-6pt}
 \caption{Large version of Fig.~\ref{fig_celeba_baselines_attributes}}
 \label{fig_celeba_baselines_attributes_large}
\end{figure}

\begin{figure}[th]
 \centering
 \includegraphics[width = 1.\textwidth]{figures/celeba_scan_traversals_large.png}
 \vspace{-6pt}
 \caption{Large version of Fig.~\ref{fig_celeba_scan_traversals}}
 \label{fig_celeba_scan_traversals_large}
\end{figure}

redpajama_set_name: RedPajamaArXiv


text:
Невский — упразднённый населённый пункт (разъезд) в Завьяловском районе Алтайского края России. Входил в состав Светловского сельсовета.

История
Упразднён согласно Закону Алтайского края от 10 июня 2009 г. № 35-ЗС «Об упразднении разъезда Невский Светловского сельсовета Завьяловского района Алтайского края и поселка Карасево Сузопского сельсовета Солтонского района Алтайского края и о внесении изменений в отдельные законы Алтайского края»

География
Разъезд находился в западной части Алтайского края, в пределах Кулундинской равнины.

Инфраструктура
Из Пояснительной записки к проекту закона Алтайского края «Об упразднении разъезда Невский Светловского сельсовета Завьяловского района Алтайского края и посёлка Карасево Сузопского сельсовета Солтонского района Алтайского края и о внесении изменений в отдельные законы Алтайского края»

По состоянию на 1 января 2009 года населения, проживающего в разъезде, нет, отсутствуют жилые помещения и постройки, что подтверждается актом комиссии, созданной на основании распоряжения главы администрации Светловского сельсовета от 10.02.2009 № 12. Вместе с тем на территории населённого пункта расположено здание бывшего железнодорожного вокзала, балансодержателем которого является Алтайское отделение структурного подразделения Западно-Сибирской железной дороги ОАО «Российские железные дороги». Железнодорожный вокзал по прямому назначению не используется. В здании находится техническое оборудование для обслуживания прилегающего железнодорожного полотна. Согласно акту об обследовании территории разъезда Невский Светловского сельсовета Завьяловского района Алтайского края оснований для восстановления населённого пункта нет, перспективы его дальнейшего развития отсутствуют. Учитывая данные обстоятельства, Совет депутатов Светловского сельсовета Завьяловского района Алтайского края обратился с ходатайством об упразднении населённого пункта разъезда Невский Светловского сельсовета Завьяловского района Алтайского края (решение от 20.03.2009 № 31).

Ссылки
http://www.geonames.org/6906996 Raz"yezd Nevskiy

Примечания

населённые пункты, упразднённые в 2009 году
Упразднённые населённые пункты Завьяловского района (Алтайский край)

redpajama_set_name: RedPajamaWikipedia


text:
AP Explains: The wide reach of Trump's Venezuela sanctions
Venezuela Sao Paulo Forum
A woman walks next to a sign that reads in Spanish "Trump, Unblock Venezuela," a reference to the Venezuelan government view that U.S. sanctions, aimed at toppling President Nicolas Maduro, are driving the country's economic problems, at the Sao Paulo Forum in Caracas, Venezuela, Friday, July 26, 2019. (AP Photo/Leonardo Fernandez)
Francisco Rodríguez
CHRISTINE ARMARIO
53rd President of Venezuela
BOGOTA, Colombia (AP) — Wide-reaching U.S. sanctions aimed at toppling Venezuelan President Nicolas Maduro's government could inflict further damage on an economy already reeling from six-digit inflation by scaring off remaining investors. The new measures don't include Venezuela's private sector, but the mere prospect of being sanctioned by the U.S. by engaging in transactions that in any way involve the Maduro administration could serve as a powerful deterrent.
Here's a look at the scope and potential impact of President Donald Trump's executive order:
IS IT AN EMBARGO?
The new sanctions announced by the Trump administration this week freeze all Venezuelan government assets in the United States and allow the Treasury Department to sanction any person, business or other entity assisting the Maduro administration.
While those measures are similar to strict U.S. sanctions on nations like North Korea, Iran and Cuba, they are not as wide in scope. Notably, Venezuela's still sizeable private sector is not blacklisted.
Francisco Rodriguez, chief economist of New York-based Torino Capital, and other analysts say they would not characterize the sanctions as an embargo since they specifically target the government and not overall trade.
But Rodriguez adds that previous U.S. sanctions nonetheless constitute an "oil embargo" since they target Venezuela's state-run oil company, which controls all transactions.
WHAT'S THE LIKELY IMPACT?
The biggest impact probably will come from "secondary sanctions" that could have a devastating effect on Venezuela's economy.
The Trump administration can now punish foreign governments and businesses that help Maduro stay in power from doing business in the U.S. National Security Adviser John Bolton put it in stark terms Tuesday: Do business with the Venezuelan government and you'll be barred from the U.S.
"Make a very careful calculation," he warned.
Analysts say the definition of providing "material support" is so vague that it could have a ripple effect that chills all business with Venezuela. In particular, countries like India and Malaysia that now buy 46 percent of Venezuela's exports could decide they are better off reducing trade.
"All these measures are impacting the economy's import capacity and will lead to a deeper economic contraction," Rodriguez said.
WILL IT BE ENFORCED?
It remains to be seen how strictly the Trump administration will enforce the new sanctions against third-party foreign entities.
Jeffrey Schott, a senior fellow at the Peterson Institute for International Economics who has extensively studied sanctions, said the onus will fall on individual companies to ensure that their transactions do not violate sanctions, and because of the Venezuelan state's wide reach, it will be difficult for many businesses to show that zero government capital is involved.
"Because the scope of the obligation is ambiguous, a company doesn't know when it is at risk of being in non-compliance," Schott said.
Rodriguez said it will be a "daunting task" for financial institutions to determine whether they are being used as a channel for government transactions, and the "prudent response for many" will be to significantly restrict such dealings.
WILL EXEMPTIONS BE ALLOWED?
The executive order allows for the delivery of humanitarian assistance, but even shipments of food and medicine are likely to diminish.
Similar sanctions imposed in other parts of the world show organizations and financial institutions typically choose to err on the side of caution even when U.S. policy explicitly allows for the delivery of certain goods.
"They usually don't continue even though they're authorized," Schott said.
He described it as an example of the "collateral damage" that sanctions often involve: Businesses and people who are not specifically targeted by the measures are nonetheless hurt.
Restrictions on trade have made many banks and companies around the world hesitant to do business with Iran, for example, even though the U.S. insists that medicine and humanitarian goods are exempt from sanctions. Prices on imported medicines like chemotherapy drugs have soared and doctors worry they will become out-of-reach for many Iranians .
ARE ECONOMIC SANCTIONS EFFECTIVE?
Studies say that at best economic sanctions are effective in only a third of cases in which they are imposed to achieve political goals.
A frequently cited review of 115 cases from 1914 to 1990 found 40 instances that could be characterized as a success. Subsequent analyses have challenged that conclusion and contend they are effective at an even lower rate.
In defending the sanctions, Bolton said: "It worked in Panama, it worked in Nicaragua once, and it will work there again, and it will work in Venezuela and Cuba."
None of those examples are clear cut, however.
U.S. sanctions on Nicaragua seeking to topple the Sandinista government in the 1980s inflicted a devastating blow on the economy that some argue led to the party's defeat in elections, but the sanctions were combined with military efforts against the regime. Panamanian strongman Manuel Noriega was only toppled when U.S. President George H.W. Bush sent in troops. Cuba remains governed by the Communist Party despite more than a half century of U.S. sanctions.
Rodriguez said the sanctions are unlikely to sway Maduro to leave the presidency. Rather, he said, high-level guarantees assuring members of the Maduro administration they won't be prosecuted if they give up power would be more effective. "You have to build that exit option," he said.

redpajama_set_name: RedPajamaCommonCrawl


In view of the gravity of the situation, the Council considers that additional persons should be added to the list of natural and legal persons, entities and bodies subject to restrictive measures as set out in Annex I to Regulation (EU) No 269/2014.
The persons listed in the Annex to this Regulation shall be added to the list set out in Annex I to Regulation (EU) No 269/2014.
Done at Brussels, 28 April 2014.
Deputy Prime Minister. Responsible for overseeing the integration of the annexed Autonomous Republic of Crimea into the Russian Federation.
Plenipotentiary Representative of the President of the Russian Federation into the so called 'Crimean Federal District', Non-permanent member of the Russian Security Council. Responsible for the implementation of the constitutional prerogatives of the Russian Head of State on the territory of the annexed Autonomous Republic of Crimea.
Minister for Crimean Affairs. Responsible for the integration of the annexed Autonomous Republic of Crimea into the Russian Federation.
Acting governor of the Ukrainian annexed city of Sevastopol.
Member of the Russian Federation Council from the annexed Autonomous Republic of Crimea.
Deputy Chairman of State Duma, United Russia — Responsible for initiating legislation to integrate the annexed Autonomous Republic of Crimea into the Russian Federation.
Deputy Chairman of State Duma, United Russia. Responsible for initiating legislation to integrate the annexed Autonomous Republic of Crimea into the Russian Federation.
Director of GRU (Main Intelligence Directorate), Deputy Chief of the General Staff of the Armed Forces of the Russian Federation, Lieutenant-General. Responsible for the activity of GRU officers in Eastern Ukraine.
Chief of the General Staff of the Armed Forces of the Russian Federation, First Deputy Minister of Defence of the Russian Federation, General of the Army. Responsible for the massive deployment of Russian troops along the border with Ukraine and lack of de-escalation of the situation.
Active leader of the 'Lugansk Guard'. Took part in the seizure of the building of the Lugansk regional office of the Security Service, recorded a video address to President Putin and Russia from the occupied building. Close links with the 'Army of the South-East'.
One of the leaders of the separatist group 'Army of the South-East' which occupied the building of the Security Service in the Lugansk region. Retired officer. Before seizing the building he and other accomplices possessed arms apparently supplied illegally from Russia and from local criminal groups.
Head of the 'Donetsk Republic', active participant and organiser of separatist actions, co-ordinator of actions of the 'Russian tourists' in Donetsk. Co-founder of a 'Civic Initiative of Donbass for the Eurasian Union'.
One of the leaders of the Donetsk People's Republic. Participated in the seizure and occupation of the regional administration. Active spokesperson for the separatists.
One of the leaders of ideologically radical organization People's Militia of Donbas. He took active part in the seizure of a number of state buildings in Donetsk region.
Identified as staff of Main Intelligence Directorate of the General Staff of the Armed Forces of the Russian Federation (GRU). He was involved in incidents in Sloviansk. He is an assistant on security issues to Sergey Aksionov, self-proclaimed prime-minister of Crimea.