text:
\section{Appendix}
\subsection{Datasets}
\subsection{dSprites and 3D Chairs}
\begin{figure}[h]
    \centering
    \subfigure[]{\includegraphics[width=0.45\linewidth]{pics/dsprites/data_samples.png}}
    \subfigure[]{\includegraphics[width=0.45\linewidth]{pics/chairs/data_samples.png}}
    \caption{Real samples from the training dataset.}
\end{figure}

\subsubsection{Designed Datasets}

\begin{figure}
\centering
    \subfigure[A1]{
    \includegraphics[width=.3\linewidth]{pics/A1_traslating.pdf}
    }
    \subfigure[A2]{
    \includegraphics[width=.3\linewidth]{pics/A2_traslating.pdf}
    }
    \subfigure[A3]{
    \includegraphics[width=.3\linewidth]{pics/A3_translating.pdf}
    }
\caption{Samples on A1, A2, A3. We visualize these samples by afterimage, and the image starts with a dark colour, then, it goes brighter as traversing the factor.}
\label{fig:translating}
\end{figure}

\subsection{Training details}
The basic architecture for all experiments follows the settings on \cite{Burgess.2018}. The hyperparameters of our proposed methods are listed in Tab. \ref{tab:setting_dsprites}. Tab. \ref{tab:threshold_chairs} and \ref{tab:threshold_dsprites} show the measured thresholds of the intrinsic action sequences.
\begin{table}[tb]
    \centering

    \begin{tabular}{|c|c|c|c|}
    \hline
    Phase   & 1             & 2             & 3             \\ \hline
     \hline
    $E_1$     & \textbf{5e-4} & 5e-5          & 5e-5          \\ \hline
    $E_2$     & 0             & \textbf{5e-4} & 5e-5          \\ \hline
    $E_3$     & 0             & 0             & \textbf{5e-4} \\ \hline
     \hline
    $\beta_{1}$ & 100           & 40            & 4             \\ \hline
    $\beta_2$ & 60           & 20            & 2             \\ \hline
    \end{tabular}

    \caption{Training settings on dSprites and 3D Chairs. $E_i$ denotes the learning rate of $i$-th group of sub-encoder. $\beta$ denotes the regularisation coefficient before the KL divergence, $\beta_1$ for dSprites, $\beta_2$ for 3D Chairs}
    \label{tab:setting_dsprites}

\end{table}

\begin{table}
\centering
\begin{minipage}[t]{0.45\linewidth}
\centering
    \begin{tabular}{|c|c|}
    \hline
      Factor      & Threshold \\ \hline
      Shape       & 32        \\ \hline
      Scale       & 100       \\ \hline
      Orientation & 5         \\ \hline
      Position  X & 120+      \\ \hline
      Position  Y & 120+      \\ \hline
    \end{tabular}
    \caption{Thresholds of actions on dSprites.}
    \label{tab:threshold_dsprites}
\end{minipage}
    \begin{minipage}[t]{0.45\linewidth}
    \centering
    \begin{tabular}{|c|c|}
    \hline
     Factor      & Threshold \\ \hline
     Chair size  & 60        \\ \hline
     Leg style  & 20       \\ \hline
     Swivel & 2         \\ \hline
     Unknown & 2         \\ \hline
    \end{tabular}
    \caption{Thresholds of actions on 3D Chairs.}
    \label{tab:threshold_chairs}
\end{minipage}
\end{table}

\subsection{Samples}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{pics/dsprites/sample_latent.png}
    \caption{Latent traversal plots for FVAE on dSprites.
    The top row show the real samples from dSprites.
}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{pics/chairs/sample_latent.png}
    \caption{Latent traversal plots for FVAE on 3D Chairs. The top row show the real samples from 3D Chairs.
}
\end{figure}
\section{Unsupervised disentanglement learning}
We first introduce the abstract concepts and the basic definitions, followed by the explanations based on information theory and other related works.
This article focuses on the explanation of information theory and the proposed models based on VAEs.
\subsection{The concept}
Disentanglement learning is fascinating and challenging because of its intrinsic similarity to human intelligence.
As depicted in the seminal paper by \citeauthor{Bengio.2013}, humans can understand and reason from a complex observation to the explanatory factors.
A common modelling assumption of disentanglement learning is that the observed data is generated by a set of ground-truth factors.
Usually, the data has a high number of dimensions, hence it is hard to understand, whereas the factors have a low number of dimensions thus simpler and easier to be understood.
The task of disentanglement learning is to uncover the ground-truth factors, such factors are invisible to the training process in an unsupervised setting.
The invisibility of factors makes it hard to define and measure disentanglement \citep{Do.2020}.

Furthermore, it is shown in \cite{Locatello.2019} that it is impossible to unsupervisedly disentangle the underlying factors for the arbitrary generative models without inductive biases.
In particular, they suggest that the inductive biases on the models and the data should be exploited.
However, they do not provide a formal definition of the inductive bias and such a definition is still unavailable.

\subsection{Information bottleneck}
Most of the dominant disentangling approaches are the variants of variational autoencoder (VAE).
The variational autoencoder (VAE) is a popular generative model, assuming that the latent variables obey a specific prior (normal distribution in practice).
The key idea of VAE is maximizing the likelihood objective by the following approximation:

\begin{equation}\label{eq:vae}
    \mathcal{L}(\theta, \phi; x,z) = \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}[\log{p_\theta (x|z)}] -
   D_{\mathrm{KL}}(q_\phi(z|x) || p(z)),
\end{equation}

which is know as the evidence lower bound (ELBO);
where the conditional probability $P(x|z),Q(z|x)$ are parameterized with deep neural networks.

\citeauthor{Higgins.2017} find that the KL term of VAEs encourages disentanglement and introduce a hyperparameter $\beta$ in front of the KL term.
They propose the \ensuremath{\beta}-VAE\xspace maximizing the following expression:
\input{formula/beta-vae}
$\beta$ controls the pressure for the posterior $Q_{\phi}(z|x)$ to match the factorized unit Gaussian prior $p(z)$.
Higher values of $\beta$ lead to lower implicit capacity of the latent information and ambiguous reconstructions.
\citeauthor{Burgess.2018} propose the Annealed-VAE that progressively increases the information capacity of the latent code while training:
\input{formula/annealed-vae}
where $\gamma$ is a large enough constant to constrain the latent information, $C$ is a value gradually increased from zero to a large number to produce high reconstruction quality.
As the total information bottleneck gradually increasing, they hypothesize that the model will allocate the capacity of the most improvement of the reconstruction log-likelihood to the encoding axes of corresponding factor.
However, they did not exploit why each factor makes the different contributions to the reconstruction log-likelihood.

\subsection{Other related work}
The other dominant direction  initiates from the prior of factors.
They assume that the ground-truth factors are independent of each other, and a series of methods enforce the latent variables have the same structure as the factors.
FactorVAE \citep{Kim.2018} applies a discriminator to approximately calculate the total correlation (TC, \cite{Watanabe.1960});
$\beta$-TCVAE \citep{Chen.2018} promotes the TC penelty by decomposing the KL term;
DIP-VAE \citep{Kumar2018DIPVAE} identifies the covariance matrix of $q(z)$.
However, the real-world problems have no strict constrain in the prior of factors.
For instance, the length of hair and the gender are two independent factors to describe a person, but the real situation is that women are more likely to have long hair.

\input{figures/projections}
\section{Disentangling action sequences}

For machines and humans, disentangling the underlying factors is a challenging task.
For instance, there are more than 1,400 breeds of dogs in the world, and it seems impossible for an ordinary person to distinguish all of them just by looking at their pictures.
The challenge in disentangling the underlying factors is mainly due to the complexity of establishing relationships without supervision, where the corresponding models should contain some level of prior knowledge or inductive biases.
However, it is possible to determine the differences without having extensive knowledge.
For example, one may mistakenly identify the breed of a dog by looking at a picture but is almost impossible to misrecognize a dog as a cat.
Therefore, in practice, discovering the differences or similarities is often a much easier task than that of uncovering the underlying factors, and it does not also need much prior knowledge.
One may conclude that discovering the commonalities between the things is an important step toward disentanglement.

\subsection{Disentangled representations}
Although the disentangling approaches can learn informative, independent, and meaningful representations on dSprites \citep{dsprites17}, these models disentangle the ground-truth factors by accident.
In other words, there is no reason for the latent variables to have any structure since there always exists a decoder that can map the latent variable into the desired output.
To determine the representations that the current approaches learn, here we design a dataset family to reveal the common aspects of these representations.

We create a toy dataset family--- each dataset contains 40x40 images which are generated from an original image of a 11x5 rectangle, by translating on a 64x64 canvas.
Each image on the dataset has a unique label (position of X, position of Y) to describe the ground-truth factors.
In this dataset family, there are two variables: the orientation of the rectangle and the way to determine the two factors.
There are infinite solutions to determine these two factors; the polar coordinate system and the Cartesian coordinate system are the most common solution.
We then create a baseline dataset, A1, with a horizontal rectangle in the Cartesian coordinate system and obtain its variants.
A2 differs in the positions determined by the polar coordinate system, and A3 differs in the orientation (45 degrees) of the images.
For the experiment settings, we choose the well-examined baseline model, \ensuremath{\beta}-VAE\xspace ($\beta=50$), and the backbone network follows the settings in \cite{Locatello.2019}.

As it is shown in Fig. 1(a), we visualize the learned representation on the latent space.
In the left column, each point represents an image with the annotation of the corresponding factors.
We also link the points with the same controlling factor with coloured lines to show the ground-truth transformation.
We argue that the factors are not the key to disentanglement since the learned representations are changed while the factors are unchanged (A1, A3), and the learned representations do not change while the factors are changed (A1, A2).
From Fig. \ref{fig:leaned_presentation}, one can see that the invariant in these three cases is that they learn a representation moving along the direction of the long side of the rectangle and the orthogonality direction.
The disentangled representations consider the rectangle as the reference system.

\subsection{Action sequences }
We assume that images of the observed data consist of a set of actions such as rotation, translation, and scaling.
Given a subset of images from original data set, we define the \textbf{meaningful action} as a sequence, i.e. an ordered permutation of elements from the subset.
To obtain such sequences, one can get an action by traversing the ground-truth factors:
\begin{equation}
    {\mathbb{S}}=\{{\bm{x}}_i\}=T(\{{\bm{c}}_i\}),
\end{equation}
where $T$ is a transformation, ${\bm{c}}_i$ is the set of factors, ${\bm{x}}_i$ is an image from the dataset.
Therefore, the disentanglement learning can be interpreted as a process of learning meaningful action sequences.
Furthermore, these actions have associated parameters to illustrate the changing processes in detail, i.e., a parameterized action sequence represents an action.
We define an action sequence as a subset of images from the dataset, and it reveals the relationships amongst the images.
The formal definition of action sequences can be described as the latent traversal in \cite{Higgins.2017}.

For an autoencoder (AE) architecture, the neural networks are flexible enough to approximate any actions.
For clarity, we define an action as a sequence of images controlled by the ground-truth factors, and an action sequence is the approximation of this action generated by a neural network.
Due to the strong expression of neural networks, the sequences may be organized in any given order.
As shown in \ref{fig:seqs}, the neural network can form any possible sequences, and among them, the ordered sequences follow human's intuition---the circle becomes larger or smaller.
Using VAEs, minimizing the objective increase the overlap between the posterior distributions across the dataset \cite{Burgess.2018}, and it then leads to disentanglement and emerges human intuition.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{pics/sequences.pdf}
    \caption{An example of action sequences. There are three images with differing sizes on the dataset (left box), and the AE learns a representation with one dimension of latent space (the double-arrow lines). So the total number of possible sequences is six. Two of them are meaningful, and the others are somewhat random.}
    \label{fig:seqs}
\end{figure}

\begin{figure}[tbp]
\centering
    \subfigure[Entropy]{
    \includegraphics[width=0.45\textwidth]{pics/A4_H}
    }
    \subfigure[KL divergence]{
    \includegraphics[width=0.45\textwidth]{pics/A4_KL}
    }
\centering
\caption{Comparison of entropy and KL divergence on A4.}
\label{fig:H_vs_KL}
\end{figure}

\subsection{Significance of sequences}
Although \cite{Locatello.2019} addresses the importance of inductive biases, few studies investigate the biases for disentanglement on the data.
As we have shown in Sec. 3.1, the orientation of the rectangle can affect the direction of disentangled representation.
We assume that there are much more inductive biases on the data which have not been yet exploited.
It is suggested in \cite{Burgess.2018} that the latent components have different contributions to the objective function.
Here, we investigate the reasons for different contributions and show that the capacity of latent information is correlated to the significance of the sequences.
To measure the significance of actions, we use the information among this action:
\begin{equation}
    \displaystyle H({\mathbb{S}}) = -\int_{\mathbb{S}} p({\bm{x}}) \mathrm{log} (p({\bm{x}}))d{\bm{x}},
\end{equation}
where ${\mathbb{S}}$ is the action controlled by the factor, .

We design a translation family (A4) with two controllable parameters, $\theta, L$ to determine the significance of the sequence; $\theta$ is the orientation of the rectangle; $L$ is the maximal distance of translating.
All datasets have only one factor which is the position of $X$-axis, and also contain one sequence.
If the value of $L$ is small enough, the images in the sequence are almost the same, this sequence is referred to as insignificant.
Both $\theta, L$ have different effects on the significance of the sequence.
As shown in Fig. \ref{fig:H_vs_KL}, the trend of KL divergence is consistent with that of entropy. Please note that the maximum for both are reached when $\theta=90$ and $L$ is at its maximum. These imply that the significance is closely related to information among the sequence.

Therefore, a higher significance of the sequence results in more information on the latent variables.
In contrast, by the gradual increase of the pressure on the KL term, the latent information decreases until it reaches zero.
One can infer that there exists a threshold of latent information.
Experimental results show a positive correlation between the significance of data and the threshold of latent information.

\section{Plain implementation according to thresholds}
\begin{figure}[ht!]
    \subfigure[FVAE]{\includegraphics[width=0.7\linewidth]{pics/architecture.pdf}
    }%
    \subfigure[Mixed label]{\includegraphics[width=.28\linewidth]{pics/labeled_test.pdf}
    }
    \caption{(a) The architecture of FVAE. Though the samples distribute in the dataset randomly, they have intrinsic significance. Under a high pressure ($\beta$), the significant actions can pass information alone the red line to itself along, and the information of insignificant actions is blocked. (b) The decoder receive the label information except for the target action.}
    \label{fig:architecture}
\end{figure}

We have discussed the situation with only one action and there are various thresholds for different significant actions.
Particularly, if $\beta$ is large enough, information of the insignificant actions will be blocked, leading to the disentangling process decays to a single factor discovery problem.
From the modelling perspective, the learning process is similar to a single action learning problem.
However, the difficulty of disentanglement is that different kinds of ground-truth actions are mixed, and a single and a fixed parameter $\beta$ is unable to separate them.
Therefore, a plain idea is to set different thresholds on the learning phases, and then in each phase, we enforce the model to learn specific actions by blocking information of the secondarily significant actions.
We propose a fractional variational autoencoder (FVAE) which disentangles the action sequences step-by-step.
The architecture of FVAE is shown in Fig. 4(a).
The encoder consists of several groups of sub-encoders, and the inputs of the decoder are the concatenated codes of all sub-encoders.
Besides, to prevent re-entangling the learned actions, we set different learning rates for the sub-encoders, which reduces the learning rate for the reset of N-1 groups and prevent the model from allocating the targeted action into the leaned codes.
The training process of FVAE is similar to the common operation in chemistry for separating mixtures---distillation.
To separate a mixture of liquid, we repeat the process of heating the liquid with different boiling points, and in each step, the heating temperature is different to ensure that only one component is being collected.

\textbf{Discussion}
Although AnnealedVAE follows the same principles as the FVAE, it differs in the interpretation of the effects of beta, and it does not explicitly prevent mixing the factors.
Moreover, the performance of AnnealedVAE depends on the choice of hyperparameter in practice \cite{Locatello.2019}.
``A large enough value'' is hard to determine, and the disentangled representation is re-entangled for an extremely large C.
To address this issue, here we introduce two methods to determine the thresholds in each phase for the labelled and unlabelled tasks.

\begin{figure}
    \centering
    \subfigure[]{
        \includegraphics[width=.45\linewidth]{pics/dsprites/threshold.pdf}
        \label{fig:threshold_dsprites}
    }
    \subfigure[]{
        \includegraphics[width=.45\linewidth]{pics/chairs/threshold.pdf}
        \label{fig:threshold_chairs}
    }

    \caption{$\beta$ vs KL divergence on dSprites (left) and 3D Chairs (right). }
    \label{fig:threshold}
\end{figure}

\begin{figure}
    \centering
    \subfigure[\ensuremath{\beta}-VAE\xspace]{
        \includegraphics[width=.23\linewidth]{pics/dsprites/betaH_gt_vs_latent.png}
    }%
    \subfigure[FVAE]{
        \includegraphics[width=.23\linewidth]{pics/dsprites/gt_vs_latent.png}
    }%
    \subfigure[Disentanglement]{
        \includegraphics[width=.48\linewidth]{pics/dsprites/mig.pdf}
    }
    \caption{Comparison between \ensuremath{\beta}-VAE\xspace and FVAE. (a) and (b) show the relationship between ${\bm{z}}$ and factors on dSprites (only active units are shown). (c) The disentanglement scores of \ensuremath{\beta}-VAE\xspace and FVAE (MIG, \cite{Chen.2018}).
}
    \label{fig:comparison}
\end{figure}

\subsection{Labelled task}

For the labelled setting, we focus on one type of actions and clip the rest of them at first.
However, the samples of one action are usually insufficient.
For example, there are only three types of shapes on dSprites.
Besides, the label information may be corrupted and only some parts of the dataset are labelled.

To address these issues, we introduce the architecture shown in Fig. 4(b), in which the label information excepted for the target actions are directly provided to the decoder.
We evaluate FVAE on dSprites dataset (involving five actions: translating, rotating, scaling, and shaping).
We first measure the threshold of each action, and the result is shown in Fig. \ref{fig:threshold_dsprites}.
One can see that the thresholds of translating and scaling are higher than the others.
This suggests that these actions are significant and easy to be disentangled.
This is in line with the results in  \cite{Burgess.2018,Higgins.2017}.

According to these thresholds, we then arrange three stages for dSprites.
At each stage, we set up a bigger number of $\beta$ than the threshold of the secondary action.
The pressure on the KL term also prevents the insignificant actions from being disentangled and ensures that the model only learns from the information of the target action.
The training of each stage can be found in the Appendix.
As shown in Fig. 7(a), the translation factor is disentangled at first easily, while it is hard to distinguish the shape, orientation, and scale.
Gradually, scaling and orientation also emerge in order.
Nevertheless, it should be noted that the shaping is still hard to be separated.
This could be attributed to the lack of commonalities between these three shapes on dSprites and motion compensation for a smooth transition.
In other words, in terms of shape, the lack of intermediate states between different shapes is an inevitable hurdle for its disentanglement.
Fig. \ref{fig:comparison} shows more substantial difference between the \ensuremath{\beta}-VAE\xspace and the FVAE.
\ensuremath{\beta}-VAE\xspace has an unstable performance compared to FVAE, and position information entangles with orientation on some dimension.

\subsection{Unlabelled task}
For the unlabelled setting, we introduce the annealing test to detect the potential components.
In the beginning, a very large value for $\beta$ is set to ensure that no action is learned.
Then, we gradually decrease $\beta$ to disentangle the significant actions.
There exists a critical point in which the latent information starts increasing and that point approximates the threshold of the corresponding action.

3D Chairs is an unlabelled dataset containing 1394 3D models from the Internet.
Fig. \ref{fig:threshold_chairs} shows the result of the annealing test on 3D Chairs.
One can recognize three points where the latent information suddenly increases: 60, 20, 4.
Therefore, we arrange a three-stage training process for 3D Chairs (more details in the Appendix).
As shown in Fig. 7(b), one can see the change of azimuth in the first stage.
In the second stage, one can see the change of size and in the third stage, one can see the change of the leg style, backrest, and material used in the making of the chair.
\input{figures/stages}

\section{Conclusion}
We demonstrated an example of the effects of images' orientation on the disentangled representations.
We have further investigated the inductive biases on the data by introducing the concept of disentangling action sequences, and we regarded that as discovering the commonality between the things, which is essential for disentanglement.
The experimental results revealed that the actions with higher significance have a larger value of thresholds of latent information.
We further proposed the fractional variational autoencoder (FVAE) to disentangle the action sequences with different significance step-by-step.
We then evaluated the performance of FVAE on dSprites and 3D Chairs.
The results suggested robust disentanglement where re-entangling is prevented.

This paper proposed a novel tool to study the inductive biases by action sequences.
However, other properties of inductive biases on the data remain to be exploited.
The current work focuses on an alternative explanation for disentanglement from the perspective of information theory. In the future, the influence of independence on disentanglement requires further investigation.

\section*{Acknowledgements}

\section{Introduction}
The basis of artificial intelligence is to understand and reason about the world based on a limited set of observations.
Unsupervised disentanglement learning is highly desirable due to its similarity with the way we as human thinking.
For instance, we can infer the movement of a running ball based on a single glance  this is because the human brain is capable of disentangling the positions from a set of images.
It has been suggested that a disentangled representation is helpful for a large variety of downstream tasks \citep{Schlkopf2012OnCA,Peters2017ElementsOC}.
According to \cite{Kim.2018},  a disentangled representation promotes interpretable semantic information.
That brings substantial advancement, including but not limited to reducing the performance gap between humans and AI approaches \citep{Lake.2017,Higgins2018SCANLH}.
Other instances of disentangled representation include semantic image understanding and generation \citep{Lample.2017, Zhu.2018GANs, Elgammal.2017}, zero-shot learning \citep{Zhu.2019}, and reinforcement learning \citep{HigginsPRMBPBBL17}.
Despite the advantageous of the disentangling representation approaches, there are still two issues to be addressed including the abstract notion and the weak explanations.

\paragraph{Notion} the conception of disentangling factors of variation is first proposed in 2013.
It is claimed in \cite{Bengio.2013} that for observations the considered factors should be explanatory and independent of each other.
The explanatory factors are however hard to formalize and measure.
An alternative way is to disentangle the ground-truth factors \citep{Ridgeway2016ASO,Do.2020}.
However, if we consider the uniqueness of the ground-truth factors, a question arise here is how to discover it from multiple equivalent representations?
As a proverb ``one cannot make bricks without straw'', \cite{Locatello.2019} prove the impossibility of disentangling factors without the help of inductive biases in the unsupervised setting.

\paragraph{Explanation} There are mainly two types of explanations for unsupervised disentanglement: information bottleneck, and independence assumption.
The ground-truth factors affect the data independently, therefore, the disentangled representations must follow the same structure.
The approaches, holding the independence assumption, encourage independence between the latent variables  \citep{Schmidhuber.1992,Chen.2018,Kim.2018,Kumar2018DIPVAE,Lopez2018InformationCO}.
However, the real-world problems have no strict constraint on the independence assumption, and the factors may be correlative.
The other explanation incorporates information theory into disentanglement.
\citeauthor{Burgess.2018,Higgins.2017,InsuJeon.2019,Saxe.2018} suggest that a limit on the capacity of the latent information channel promotes disentanglement by enforcing the model to acquire the most significant latent representation.
They further hypothesize that the information bottleneck enforces the model to find the significant improvement.

In this paper, we first demonstrate that instead of the ground-truth factors the disentangling approaches learn actions of translating based on the orientation of the images.
We then propose the concept of disentangling actions which discover the commonalities between the images and categorizes them into sequences.
We treat disentangling action sequences as a necessary step toward disentangling factors, which can capture the internal relationships between the data, and make it possible to analyze the inductive biases from the data perspective.
Furthermore, the results on a toy example show that the significance of actions is positively correlated with the threshold of latent information.
Then, we promote that conclusion to complex problems.
Our contributions are summarized in the following:
\begin{itemize}
    \item We show that the significance of action is related to the capacity of learned latent information, resulting in the different thresholds of factors.
    \item We propose a novel framework, fractional variational autoencoder (FVAE) to extracts explanatory action sequences step-by-step, and at each step, it learns specific actions by blocking others' information.
\end{itemize}

We organize the rest of this paper as follows.
Sec.2 describes the development of unsupervised disentanglement learning and the proposed methods based on VAEs.
In Sec.3, through an example, we show that the disentangled representations are relative to the data itself and further introduce a novel concept, i.e., disentangling action sequences.
Then, we investigate the inductive biases on the data and find that the significant action has a high threshold of latent information.
In Sec.4, we propose a step-by-step disentangling framework, namely fractional VAE (FVAE), to disentangle action sequences.
For the labelled and unlabelled tasks, we respectively introduce two methods to measure their thresholds.
We then evaluate FVAE on a labelled dataset (dSprites, \cite{dsprites17}) and an unlabelled dataset (3D Chairs, \cite{Aubry2014Chairs}).
Finally, we conclude the paper and discuss the future work in Sec.5
\section{Unsupervised disentanglement learning}
We first introduce the abstract concepts and the basic definitions, followed by the explanations based on information theory and other related works.
This article focuses on the explanation of information theory and the proposed models based on VAEs.
\subsection{The concept}
Disentanglement learning is fascinating and challenging because of its intrinsic similarity to human intelligence.
As depicted in the seminal paper by \citeauthor{Bengio.2013}, humans can understand and reason from a complex observation to the explanatory factors.
A common modelling assumption of disentanglement learning is that the observed data is generated by a set of ground-truth factors.
Usually, the data has a high number of dimensions, hence it is hard to understand, whereas the factors have a low number of dimensions thus simpler and easier to be understood.
The task of disentanglement learning is to uncover the ground-truth factors, such factors are invisible to the training process in an unsupervised setting.
The invisibility of factors makes it hard to define and measure disentanglement \citep{Do.2020}.

Furthermore, it is shown in \cite{Locatello.2019} that it is impossible to unsupervisedly disentangle the underlying factors for the arbitrary generative models without inductive biases.
In particular, they suggest that the inductive biases on the models and the data should be exploited.
However, they do not provide a formal definition of the inductive bias and such a definition is still unavailable.

\subsection{Information bottleneck}
Most of the dominant disentangling approaches are the variants of variational autoencoder (VAE).
The variational autoencoder (VAE) is a popular generative model, assuming that the latent variables obey a specific prior (normal distribution in practice).
The key idea of VAE is maximizing the likelihood objective by the following approximation:

\begin{equation}\label{eq:vae}
    \mathcal{L}(\theta, \phi; x,z) = \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}[\log{p_\theta (x|z)}] -
   D_{\mathrm{KL}}(q_\phi(z|x) || p(z)),
\end{equation}

which is know as the evidence lower bound (ELBO);
where the conditional probability $P(x|z),Q(z|x)$ are parameterized with deep neural networks.

\citeauthor{Higgins.2017} find that the KL term of VAEs encourages disentanglement and introduce a hyperparameter $\beta$ in front of the KL term.
They propose the \ensuremath{\beta}-VAE\xspace maximizing the following expression:
\input{formula/beta-vae}
$\beta$ controls the pressure for the posterior $Q_{\phi}(z|x)$ to match the factorized unit Gaussian prior $p(z)$.
Higher values of $\beta$ lead to lower implicit capacity of the latent information and ambiguous reconstructions.
\citeauthor{Burgess.2018} propose the Annealed-VAE that progressively increases the information capacity of the latent code while training:
\input{formula/annealed-vae}
where $\gamma$ is a large enough constant to constrain the latent information, $C$ is a value gradually increased from zero to a large number to produce high reconstruction quality.
As the total information bottleneck gradually increasing, they hypothesize that the model will allocate the capacity of the most improvement of the reconstruction log-likelihood to the encoding axes of corresponding factor.
However, they did not exploit why each factor makes the different contributions to the reconstruction log-likelihood.

\subsection{Other related work}
The other dominant direction  initiates from the prior of factors.
They assume that the ground-truth factors are independent of each other, and a series of methods enforce the latent variables have the same structure as the factors.
FactorVAE \citep{Kim.2018} applies a discriminator to approximately calculate the total correlation (TC, \cite{Watanabe.1960});
$\beta$-TCVAE \citep{Chen.2018} promotes the TC penelty by decomposing the KL term;
DIP-VAE \citep{Kumar2018DIPVAE} identifies the covariance matrix of $q(z)$.
However, the real-world problems have no strict constrain in the prior of factors.
For instance, the length of hair and the gender are two independent factors to describe a person, but the real situation is that women are more likely to have long hair.

\input{figures/projections}
\section{Disentangling action sequences}

For machines and humans, disentangling the underlying factors is a challenging task.
For instance, there are more than 1,400 breeds of dogs in the world, and it seems impossible for an ordinary person to distinguish all of them just by looking at their pictures.
The challenge in disentangling the underlying factors is mainly due to the complexity of establishing relationships without supervision, where the corresponding models should contain some level of prior knowledge or inductive biases.
However, it is possible to determine the differences without having extensive knowledge.
For example, one may mistakenly identify the breed of a dog by looking at a picture but is almost impossible to misrecognize a dog as a cat.
Therefore, in practice, discovering the differences or similarities is often a much easier task than that of uncovering the underlying factors, and it does not also need much prior knowledge.
One may conclude that discovering the commonalities between the things is an important step toward disentanglement.

\subsection{Disentangled representations}
Although the disentangling approaches can learn informative, independent, and meaningful representations on dSprites \citep{dsprites17}, these models disentangle the ground-truth factors by accident.
In other words, there is no reason for the latent variables to have any structure since there always exists a decoder that can map the latent variable into the desired output.
To determine the representations that the current approaches learn, here we design a dataset family to reveal the common aspects of these representations.

We create a toy dataset family--- each dataset contains 40x40 images which are generated from an original image of a 11x5 rectangle, by translating on a 64x64 canvas.
Each image on the dataset has a unique label (position of X, position of Y) to describe the ground-truth factors.
In this dataset family, there are two variables: the orientation of the rectangle and the way to determine the two factors.
There are infinite solutions to determine these two factors; the polar coordinate system and the Cartesian coordinate system are the most common solution.
We then create a baseline dataset, A1, with a horizontal rectangle in the Cartesian coordinate system and obtain its variants.
A2 differs in the positions determined by the polar coordinate system, and A3 differs in the orientation (45 degrees) of the images.
For the experiment settings, we choose the well-examined baseline model, \ensuremath{\beta}-VAE\xspace ($\beta=50$), and the backbone network follows the settings in \cite{Locatello.2019}.

As it is shown in Fig. 1(a), we visualize the learned representation on the latent space.
In the left column, each point represents an image with the annotation of the corresponding factors.
We also link the points with the same controlling factor with coloured lines to show the ground-truth transformation.
We argue that the factors are not the key to disentanglement since the learned representations are changed while the factors are unchanged (A1, A3), and the learned representations do not change while the factors are changed (A1, A2).
From Fig. \ref{fig:leaned_presentation}, one can see that the invariant in these three cases is that they learn a representation moving along the direction of the long side of the rectangle and the orthogonality direction.
The disentangled representations consider the rectangle as the reference system.

\subsection{Action sequences }
We assume that images of the observed data consist of a set of actions such as rotation, translation, and scaling.
Given a subset of images from original data set, we define the \textbf{meaningful action} as a sequence, i.e. an ordered permutation of elements from the subset.
To obtain such sequences, one can get an action by traversing the ground-truth factors:
\begin{equation}
    {\mathbb{S}}=\{{\bm{x}}_i\}=T(\{{\bm{c}}_i\}),
\end{equation}
where $T$ is a transformation, ${\bm{c}}_i$ is the set of factors, ${\bm{x}}_i$ is an image from the dataset.
Therefore, the disentanglement learning can be interpreted as a process of learning meaningful action sequences.
Furthermore, these actions have associated parameters to illustrate the changing processes in detail, i.e., a parameterized action sequence represents an action.
We define an action sequence as a subset of images from the dataset, and it reveals the relationships amongst the images.
The formal definition of action sequences can be described as the latent traversal in \cite{Higgins.2017}.

For an autoencoder (AE) architecture, the neural networks are flexible enough to approximate any actions.
For clarity, we define an action as a sequence of images controlled by the ground-truth factors, and an action sequence is the approximation of this action generated by a neural network.
Due to the strong expression of neural networks, the sequences may be organized in any given order.
As shown in \ref{fig:seqs}, the neural network can form any possible sequences, and among them, the ordered sequences follow human's intuition---the circle becomes larger or smaller.
Using VAEs, minimizing the objective increase the overlap between the posterior distributions across the dataset \cite{Burgess.2018}, and it then leads to disentanglement and emerges human intuition.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{pics/sequences.pdf}
    \caption{An example of action sequences. There are three images with differing sizes on the dataset (left box), and the AE learns a representation with one dimension of latent space (the double-arrow lines). So the total number of possible sequences is six. Two of them are meaningful, and the others are somewhat random.}
    \label{fig:seqs}
\end{figure}

\begin{figure}[tbp]
\centering
    \subfigure[Entropy]{
    \includegraphics[width=0.45\textwidth]{pics/A4_H}
    }
    \subfigure[KL divergence]{
    \includegraphics[width=0.45\textwidth]{pics/A4_KL}
    }
\centering
\caption{Comparison of entropy and KL divergence on A4.}
\label{fig:H_vs_KL}
\end{figure}

\subsection{Significance of sequences}
Although \cite{Locatello.2019} addresses the importance of inductive biases, few studies investigate the biases for disentanglement on the data.
As we have shown in Sec. 3.1, the orientation of the rectangle can affect the direction of disentangled representation.
We assume that there are much more inductive biases on the data which have not been yet exploited.
It is suggested in \cite{Burgess.2018} that the latent components have different contributions to the objective function.
Here, we investigate the reasons for different contributions and show that the capacity of latent information is correlated to the significance of the sequences.
To measure the significance of actions, we use the information among this action:
\begin{equation}
    \displaystyle H({\mathbb{S}}) = -\int_{\mathbb{S}} p({\bm{x}}) \mathrm{log} (p({\bm{x}}))d{\bm{x}},
\end{equation}
where ${\mathbb{S}}$ is the action controlled by the factor, .

We design a translation family (A4) with two controllable parameters, $\theta, L$ to determine the significance of the sequence; $\theta$ is the orientation of the rectangle; $L$ is the maximal distance of translating.
All datasets have only one factor which is the position of $X$-axis, and also contain one sequence.
If the value of $L$ is small enough, the images in the sequence are almost the same, this sequence is referred to as insignificant.
Both $\theta, L$ have different effects on the significance of the sequence.
As shown in Fig. \ref{fig:H_vs_KL}, the trend of KL divergence is consistent with that of entropy. Please note that the maximum for both are reached when $\theta=90$ and $L$ is at its maximum. These imply that the significance is closely related to information among the sequence.

Therefore, a higher significance of the sequence results in more information on the latent variables.
In contrast, by the gradual increase of the pressure on the KL term, the latent information decreases until it reaches zero.
One can infer that there exists a threshold of latent information.
Experimental results show a positive correlation between the significance of data and the threshold of latent information.

\section{Plain implementation according to thresholds}
\begin{figure}[ht!]
    \subfigure[FVAE]{\includegraphics[width=0.7\linewidth]{pics/architecture.pdf}
    }%
    \subfigure[Mixed label]{\includegraphics[width=.28\linewidth]{pics/labeled_test.pdf}
    }
    \caption{(a) The architecture of FVAE. Though the samples distribute in the dataset randomly, they have intrinsic significance. Under a high pressure ($\beta$), the significant actions can pass information alone the red line to itself along, and the information of insignificant actions is blocked. (b) The decoder receive the label information except for the target action.}
    \label{fig:architecture}
\end{figure}

We have discussed the situation with only one action and there are various thresholds for different significant actions.
Particularly, if $\beta$ is large enough, information of the insignificant actions will be blocked, leading to the disentangling process decays to a single factor discovery problem.
From the modelling perspective, the learning process is similar to a single action learning problem.
However, the difficulty of disentanglement is that different kinds of ground-truth actions are mixed, and a single and a fixed parameter $\beta$ is unable to separate them.
Therefore, a plain idea is to set different thresholds on the learning phases, and then in each phase, we enforce the model to learn specific actions by blocking information of the secondarily significant actions.
We propose a fractional variational autoencoder (FVAE) which disentangles the action sequences step-by-step.
The architecture of FVAE is shown in Fig. 4(a).
The encoder consists of several groups of sub-encoders, and the inputs of the decoder are the concatenated codes of all sub-encoders.
Besides, to prevent re-entangling the learned actions, we set different learning rates for the sub-encoders, which reduces the learning rate for the reset of N-1 groups and prevent the model from allocating the targeted action into the leaned codes.
The training process of FVAE is similar to the common operation in chemistry for separating mixtures---distillation.
To separate a mixture of liquid, we repeat the process of heating the liquid with different boiling points, and in each step, the heating temperature is different to ensure that only one component is being collected.

\textbf{Discussion}
Although AnnealedVAE follows the same principles as the FVAE, it differs in the interpretation of the effects of beta, and it does not explicitly prevent mixing the factors.
Moreover, the performance of AnnealedVAE depends on the choice of hyperparameter in practice \cite{Locatello.2019}.
``A large enough value'' is hard to determine, and the disentangled representation is re-entangled for an extremely large C.
To address this issue, here we introduce two methods to determine the thresholds in each phase for the labelled and unlabelled tasks.

\begin{figure}
    \centering
    \subfigure[]{
        \includegraphics[width=.45\linewidth]{pics/dsprites/threshold.pdf}
        \label{fig:threshold_dsprites}
    }
    \subfigure[]{
        \includegraphics[width=.45\linewidth]{pics/chairs/threshold.pdf}
        \label{fig:threshold_chairs}
    }

    \caption{$\beta$ vs KL divergence on dSprites (left) and 3D Chairs (right). }
    \label{fig:threshold}
\end{figure}

\begin{figure}
    \centering
    \subfigure[\ensuremath{\beta}-VAE\xspace]{
        \includegraphics[width=.23\linewidth]{pics/dsprites/betaH_gt_vs_latent.png}
    }%
    \subfigure[FVAE]{
        \includegraphics[width=.23\linewidth]{pics/dsprites/gt_vs_latent.png}
    }%
    \subfigure[Disentanglement]{
        \includegraphics[width=.48\linewidth]{pics/dsprites/mig.pdf}
    }
    \caption{Comparison between \ensuremath{\beta}-VAE\xspace and FVAE. (a) and (b) show the relationship between ${\bm{z}}$ and factors on dSprites (only active units are shown). (c) The disentanglement scores of \ensuremath{\beta}-VAE\xspace and FVAE (MIG, \cite{Chen.2018}).
}
    \label{fig:comparison}
\end{figure}

\subsection{Labelled task}

For the labelled setting, we focus on one type of actions and clip the rest of them at first.
However, the samples of one action are usually insufficient.
For example, there are only three types of shapes on dSprites.
Besides, the label information may be corrupted and only some parts of the dataset are labelled.

To address these issues, we introduce the architecture shown in Fig. 4(b), in which the label information excepted for the target actions are directly provided to the decoder.
We evaluate FVAE on dSprites dataset (involving five actions: translating, rotating, scaling, and shaping).
We first measure the threshold of each action, and the result is shown in Fig. \ref{fig:threshold_dsprites}.
One can see that the thresholds of translating and scaling are higher than the others.
This suggests that these actions are significant and easy to be disentangled.
This is in line with the results in  \cite{Burgess.2018,Higgins.2017}.

According to these thresholds, we then arrange three stages for dSprites.
At each stage, we set up a bigger number of $\beta$ than the threshold of the secondary action.
The pressure on the KL term also prevents the insignificant actions from being disentangled and ensures that the model only learns from the information of the target action.
The training of each stage can be found in the Appendix.
As shown in Fig. 7(a), the translation factor is disentangled at first easily, while it is hard to distinguish the shape, orientation, and scale.
Gradually, scaling and orientation also emerge in order.
Nevertheless, it should be noted that the shaping is still hard to be separated.
This could be attributed to the lack of commonalities between these three shapes on dSprites and motion compensation for a smooth transition.
In other words, in terms of shape, the lack of intermediate states between different shapes is an inevitable hurdle for its disentanglement.
Fig. \ref{fig:comparison} shows more substantial difference between the \ensuremath{\beta}-VAE\xspace and the FVAE.
\ensuremath{\beta}-VAE\xspace has an unstable performance compared to FVAE, and position information entangles with orientation on some dimension.

\subsection{Unlabelled task}
For the unlabelled setting, we introduce the annealing test to detect the potential components.
In the beginning, a very large value for $\beta$ is set to ensure that no action is learned.
Then, we gradually decrease $\beta$ to disentangle the significant actions.
There exists a critical point in which the latent information starts increasing and that point approximates the threshold of the corresponding action.

3D Chairs is an unlabelled dataset containing 1394 3D models from the Internet.
Fig. \ref{fig:threshold_chairs} shows the result of the annealing test on 3D Chairs.
One can recognize three points where the latent information suddenly increases: 60, 20, 4.
Therefore, we arrange a three-stage training process for 3D Chairs (more details in the Appendix).
As shown in Fig. 7(b), one can see the change of azimuth in the first stage.
In the second stage, one can see the change of size and in the third stage, one can see the change of the leg style, backrest, and material used in the making of the chair.
\input{figures/stages}

\section{Conclusion}
We demonstrated an example of the effects of images' orientation on the disentangled representations.
We have further investigated the inductive biases on the data by introducing the concept of disentangling action sequences, and we regarded that as discovering the commonality between the things, which is essential for disentanglement.
The experimental results revealed that the actions with higher significance have a larger value of thresholds of latent information.
We further proposed the fractional variational autoencoder (FVAE) to disentangle the action sequences with different significance step-by-step.
We then evaluated the performance of FVAE on dSprites and 3D Chairs.
The results suggested robust disentanglement where re-entangling is prevented.

This paper proposed a novel tool to study the inductive biases by action sequences.
However, other properties of inductive biases on the data remain to be exploited.
The current work focuses on an alternative explanation for disentanglement from the perspective of information theory. In the future, the influence of independence on disentanglement requires further investigation.

\section{Method}
\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{pics/architecture.pdf}
    \caption{Architecture of DVAE. The blue one is active.}
    \label{fig:architecture}
\end{figure}
Based on the analysis of transformations, we argue that a model has a preferred transformation.
A proper model will be easy to uncover the intrinsic relationships among the dataset and organize them into parameterized transformational sequences.
The difficulty of this task is that kinds of ground-truth transformations are mixed into one dataset, and it is hard to extract them simultaneously.
One solution is to design plenty of diverse models preferring corresponding transformations.
Thereby, we propose the fractional VAE framework based on beta-VAE as shown in Fig \ref{fig:architecture}, which has several groups of encoders with different values of beta, and the training process is similar to fractional---separating components of a mixture based on different boiling points.
We can separate the training process into several phases, and each phase has a target transformation which learned by a group of the encoder.
For phase i, we concatenate the codes of the first i-ths groups of encoders and the reset codes are sampled from Gaussian noise.
The codes of the first i-1-ths groups of encoders are learned transformations, and we hope they are stable in the rest training process.
Therefore, we give a smaller learning rate on the learned encoders to obtain a slight modification.

\subsubsection{Week supervised method for labeled tasks}
For the dataset with label, we could do a model selection process with few samples.
...some details.
From Fig \ref{fig:mig}, we find that the model with $\beta=4$ is suitable for both the translation and scaling transformations, and the model can no longer learn scaling when $\beta>12$.
The other transformations (shaping and rotation) are hard to learn.
Therefore, we apply a two-phase training process to extract fundamental transformations, and the training parameters are shown in \ref{tab:training}.

\begin{table}[tb]
    \centering
    \begin{tabular}{c|c|c|c|c}
        Phase & Lr1 & Lr2  & $\beta_1$ & $\beta_2$\\
        1 &   1e-4 & 0     &8          & 4\\
        1 &   1e-5 & 1e-4  &4          & 4\\
    \end{tabular}
    \caption{Training details. The down script denotes the group of FVAE.}
    \label{tab:training}
\end{table}

\begin{figure}[tbp]
\centering
\subfigure[$\beta$ vs. MIG]{
\begin{minipage}[t]{0.3\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/supervised/modelselection.png}
\label{fig:mig}
\end{minipage}%
}%
\subfigure[Leaned sequences on each phase]{
\begin{minipage}[t]{0.6\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/supervised/phase_dsprites.png}
\end{minipage}%
}%
\centering
\label{fig:dsprites}
\caption{Learned sequence on Dsprites.}
\end{figure}

\subsection{Annealing FVAE for unlabeled tasks}
For the unlabeled dataset, though we can not measure the suitable $\beta$, we can estimate it by changing the $\beta$ from a very high value to a small value.

\begin{figure}[tbp]
\centering
\subfigure[iterations vs. KL ]{
\begin{minipage}[t]{0.5\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/unsupervised/AnnealingTest_dsprites.png}
\end{minipage}%
}%
\subfigure[Ground-truth factors vs. latent varables]{
\begin{minipage}[t]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/unsupervised/gtvslv_dsprites.png}
\end{minipage}%
}%

\centering
\label{fig:dsprites}
\caption{Annealing FVAE on Dsprites.}
\end{figure}
\section{Disentangling Transformation}
While a series of approaches obtain some exciting results on the both artificial and real-world datasets (Dsprites,CeleBA, chairs) by enforcing the independence on latent variables, there is still a gap between the success and theorical explanation.
Because (XX) prove the impossibility of unsupervised disentanglement learning.
In this paper, we challenge the original conception of disentangling factors on disentanglement learning and provide a novel perspective about disentanglement learning---the goal is to learn a set of transformations, which revealing the internal relationship between images from data.

To prove that the disentangled representations are not relative to the ground-truth factors, we build a toy dataset family, and each dataset contains 40x40 points from one original image by translating.
In the dataset family, there are two variables: the original image and the way to determine each point by the two factors.
Every image on the dataset has an unique label (posX, posY) to describe the generative factors.
There are infinite solutions to choose these two factors, and the polar coordinate system or the Cartesian coordinate system may be the most common and well-known.
These two coordinate systems satisfy the independence condition on the factors, rising the doubt which coordinate system is preferred by the methods holding the independence assumption.

Therefore, we get a baseline dataset A1 with a 20x20 horizontal original image in the Cartesian coordinate system, A2 in the polar coordinate system, and A3 with a rotated image by 45 degrees.
A1 and A2 have the same original image but differ in factors.
A1 and A3 have the same factors but different original images.
For the experiment settings, we apply a standard architecture described in [] and the practical VAE variant beta-vae (beta=50).

\begin{figure}[tbp]
\centering
\subfigure[A1]{
\begin{minipage}[t]{0.3\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/t=0.png}
\end{minipage}%
}%
\subfigure[A2]{
\begin{minipage}[t]{0.3\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/A2.png}
\end{minipage}%
}%
\subfigure[A3]{
\begin{minipage}[t]{0.3\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/t=45.png}
\end{minipage}
}%

\centering
\label{fig:projection}
\caption{Projection on Latent Space}
\end{figure}
As shown in Fig \ref{fig:projection}, we visualize the learned representation on the latent space, and each point represents an image with the annotation of corresponding factors.
We also link the images having the same controlling factor with colored lines to show the ground-truth transformation.
We argue that factors are not the key to disentangle, since the learned representations change when the factors are unchanged (A1, A3) and the learned representations don't change when the factors are changed (A1, A2).
The factors have few effects on the learned representations.
However, our model learns the same transformation, which moves image along the direction of the rectangle.

\subsection{The impedance of Randomness}
\begin{figure}[tbp]
\centering
\subfigure[General transformations]{
\begin{minipage}[t]{0.3\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/transformation.png}
\label{fig:T}
\end{minipage}%
}%
\subfigure[Learning curves]{
\begin{minipage}[t]{0.6\linewidth}
\centering
\includegraphics[width=\textwidth]{pics/impedance.png}
\end{minipage}
}%

\centering
\label{fig:impedance}
\caption{Learning curves for the reconstruction task using: translating along y axis, x axis, and diagonal, rotating, and moving randomly. Meaningful transformations result in much faster convergence.
}
\end{figure}

As image prior shows, a neural network can recover any images, including a random noise.
However, naturally-looking images are preferred by the network, and it is much quicker to learn these images comparing to the noisy one.
One may wonder whether a decoder has a similar property--- the decoder offers high impedance to a random transformation and low impedance to a meaningful transformation.

Figure shows the reconstruction loss of the VAE as a function of the gradient descent iterations for four different choices for the transformational sequence: 1) a moving sequence along the y-axis; 2) a muddled sequence with the same images as 1); 3) a moving sequence along the x-axis; 3) a randomly selected images sequence.
One can see that the model has an inductive bias on data to learn the translation along the direction of the long side of a rectangle.
Moreover, a neural network has an intrinsic impedance to the disordered, meaningless sequence.
The images on the disordered sequence share little information.
In contrast, the images on the ordered sequence have not only meaningful information on the human view but also a lower distance on the distribution.
Thus, though a flexible neural network can build a map between the randomly selected images and an ordered sequence, it does so very reluctantly and prefers the related images.

\subsection{Threshold of Transformations}
Different transformations represent different functions, which are distinct for their complexities.
When our model learns these transformations, the learning procedure shows distinct behaviors.
As depicted in Image Prior, the model is harder to learn a random noise.
Our model shows similar properties.
Leaning a random transformation is harder than a meaningful transformation.
Different transformations also have stratified learning hardness, which provides clues for disentangling.

We compared six transformations.
We create six sub-datasets with only one corresponding transformation.
0: y
1: x
2: diagonal
3: cycle
4: rotation
5: random translation
We push different pressures on our model by setting the beta term on btcvae.

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{pics/threshold.png}
    \caption{Transformations respectively are y, x, diagonal, cycle, rotation, random translation. The color varies from red to yellow regarding 1 to 90.}
    \label{fig:threshold}
\end{figure}

The higher beta means higher pressure and leads to less information ob latent variables.
A threshold of a transformation is a critical value for learning such a transformation with the highest pressure.
From the figure \ref{fig:threshold}, we can see that the threshold of rotation is between 30~60, while others are between 60~90.
That means the rotation transformation is harder to learn.
Furthermore, based on the difference of thresholds, our model can distinguish these transformations.

redpajama_set_name: RedPajamaArXiv