text:
\section{Introduction}
Crossover designs are widely used in  pharmaceutical and clinical trials, bioequivalence studies and biological assays, where the response is quite frequently non-normal (\cite{Layard_1978}, \cite{Forster1992}, \cite{Waterhouse_2006} and \cite{Bandyopadhyay2009}) and have to be modeled using a generalized linear model  (GLM). While methods for analyzing GLM data arising from crossover trials are available in  \cite{Senn_2002} and \cite{Jones_2014}, the question of designing such studies in an optimal manner does not seem to have been addressed before in the statistical literature.
The usual practice is to extend the use of the same designs which are found to be optimal in the normal case, to these non-normal response situations.

 For illustration
consider an experiment where the experimenter is interested to study the effect of four treatments ($A,B,C,D$) on 80 subjects  in four time periods. The response variable is binary in nature. The experimenter selects the four treatment sequences \{$ABCD, BDAC, CADB, DCBA$\} forming a Williams design. For normal responses in a $4\times 4$ crossover trial, a Williams designs has been shown to be optimal, but how does the experimenter know it is also the best design when the response is binary?

In this article, we study optimal crossover designs  for GLMs.  Three case studies based on non-normal responses are used to illustrate the proposed methodology. Crossover trials with $t$ treatments and $p$ periods, for $t\leq p$ are considered. The designs proposed in this paper minimize the log determinant of the variance of the estimator of treatment contrast of direct effects over all possible allocation of the $n$ subjects to the treatment sequences. While analyzing data from  crossover trials, the correlation between observations within subjects are modeled using a ``working correlation structure", which may be  assumed to be compound symmetric or auto regressive in nature. Since the main interest is in estimating the treatment effects, the subject effects as taken as  nuisance parameters and generalized estimating equations of Liang and Zeger (1986) are used to estimate the marginal means.

As in all GLM designs, the variance of the treatment effect estimator depends on the model parameters. To address the  issue of the parameter dependence and obtain robust designs we propose
D-optimal Bayesian crossover designs. In our approach, a prior distribution is assumed on the model parameters, which is then incorporated into an appropriate objective function
(variance of the treatment contrast) by integrating and averaging over the prior distribution.
Similar to our Bayesian design criterion, an average criterion called $A$-criterion have been used before for  crossover designs for normal responses by (\cite{Kempton2001}, \cite{Baily2006}, \cite{Zheng2013} and \cite{Li2015}). Bayesian designs have been a popular choice whenever the variance-covariance matrix depends on the model parameters, for some references see  (\cite{Chaloner1989}, \cite{Dette1994}, \cite{Woods2011} and \cite{Mylona2014} ).

\section{Case studies}
For illustration purpose we consider three case studies based on crossover trials  involving binary, count and  Gamma responses.
\subsection{A four periods four treatments binary response crossover trial}
The first case study presented here is from a trial based on the four-period, four treatment
Williams design. It has been reported in  \cite{Kenward1992}. The four treatments  are denoted by A, B, C and D. Eighty subjects are randomly assigned to the four treatment sequences \{$ABCD, BDAC, CADB, DCBA$\}, with about twenty subjects allocated to each treatment sequence. The response is a binary outcome taking values 1 and 0 based on patient relief and no relief, respectively.

The research question which arises from the above case study is why did the experimenter select the 4 treatment sequences \{$ABCD, BDAC, CADB, DCBA$\} forming a Williams design (\cite{Williams1949}).  Is this the best possible selection of treatment sequences? The book by \cite{Bose2009}, page 40 shows that for normal response crossover models, for the 4 treatment and 4 periods case, Williams design is the optimal design.  But how can we be sure that the same design applies to a binary response crossover framework as well? Does the selected design change if the correlation structure between observations change say, from equicorrelated to auto regressive structure?

\subsection{Two periods two treatments Poisson response crossover trial}
This study is based on an example described in \cite{Layard_1978}. Two  drugs, standard drug A and an innovation  drug B, is administered for controlling angina in 20 patients. It is known that the innovative drug $B$ is no worse than the standard drug $A$. For a given patient, number of angina attacks on weekly basis is assumed to follow a Poisson distribution (\cite{Layard_1978}). Number of attacks for each patient of consecutive two weeks are recorded. Treatment sequences considered are \{$AB$, $BA$)\} and 10 patients are assigned to each of the treatment sequences. This is a 2-treatments 2-periods crossover trial.

As in case study I, the question arises how does the experimenter choose the design $AB, BA$. Is this the best or most efficient design under the repeated measures setup when responses follow a Poisson distribution?

\subsection{Three periods two treatments Gamma response trial}
The length of hospital stay is an important measure of the success of hospital activity, costs incurred by patients and the treatment administered to a patient. However, its empirical distribution is often right skewed and a Gamma distribution with a log link has been seen to be a good fit (\cite{Faddy2009}). In this case study we consider a crossover trial where two treatments are applied over three periods and length of hospital stay, assumed to having a Gamma distribution, is the primary end point.

As in the earlier two case studies, we investigate the best design for a two treatment three periods design with a gamma response.

\section{The model}

We consider experiments where there are $t$ treatments and $n$ subjects, and $p$ repeated measurements are taken from each subject.  The observations from each subject may be correlated. The marginal distribution of the response $Y_{ij}$  is described by a working generalized linear model with the following three components (\cite{Liang1986}):
\begin{enumerate}
\item  $Y_{ij}$ has a distribution from the exponential family form,
\begin{equation}
  f(y_{ij}|\phi_{ij},\phi)  = \mbox{exp}\left\{ \left[y_{ij}\phi_{ij} - b(\phi_{ij}) + c(y_{ij})\right]\psi + d(y_{ij}, \psi) \right\}\label{efd1}
\end{equation}
where $b(\cdot)$, $c(\cdot)$ and $d(\cdot)$ are known functions and $\psi$ is the dispersion parameter.
It can be shown that:
${E}(Y_{ij})=\mu_{ij} = \frac{db(\phi_{ij})}{d\phi_{ij}}$ and
${Var}(Y_{ij})=\frac{d^2 b(\phi_{ij})}{d\phi^{2}_{ij}}$.
\item The linear predictor $\eta_{ij}$ in a repeated measures setup can be written as (\cite{Bose2009}),
\begin{equation}\label{margmodel}
\eta_{ij}=\mu+\beta_i+\tau_{d(i,j)}+\rho_{d(i-1,j)};\,i=1\,\ldots,p,\,j=1,\ldots,n,
\end{equation}
where
$\beta_i$ represents the effect of the $i$th period, $\tau_s$ is the direct effect due to treatment $s$ and $\rho_s$ is the carryover effect due to treatment $s$,  $s=1,\ldots ,t$.
\item The mean of $y_{ij}$ denoted by $\mu_{ij}$ is related to $\eta_{ij}$ through a link function $g$, where $g(\mu_{ij})=\eta_{ij}$ and the inverse of $g$ exists.
\end{enumerate}
\subsection{Estimation}
 Regression coefficients as well as their variances are estimated by the GEE approach of \cite{Liang1986} and \cite{Zeger1988}.
Due to observations from the same subject being correlated, a ``working correlation" matrix, $R(\alpha)$, is used to describe the dependencies between repeated observations from a subject. Here, $\alpha$ is a vector of length $s$. For cases where $R(\alpha)$ is the true correlation matrix of $Y_j = (Y_{1j}, \cdots, Y_{pj})'$, the covariance of $Y_j$ is
\begin{equation}
V_j=A_j^{1/2}R(\alpha)A_j^{1/2},
\end{equation}
$A_j=\text{diag}(\text{Var}(Y_{1j}),\ldots,\text{Var}(Y_{pj}))$ and $\text{Var}(Y_{ij})$ denotes the variance of $Y_{ij}$.
Also, the asymptotic variance for the GEE estimator $\hat{\thetav}$ (see \cite{Zeger1988}, equation (3.2)) is
\begin{equation}
Var(\hat{\thetav})=
\left[\sum_{j=1}^{n}\frac{\partial\mu'_j}{\partial\thetav} V_{j}^{-1}\frac{\partial\mu_j}{\partial\thetav}\right]^{-1},
\end{equation}
where $\thetav = (\mu, \betav', \tauv', \gammav')'$, $\betav' = (\beta_{1},\cdots\beta_{p})$, $\tauv' = (\tau_{1}, \cdots, \tau_{t})$ and $\gammav' = (\gamma_{1}, \cdots, \gamma_{t})$.

 However, if the true correlation structure varies from the ``working correlation" structure, then $Var(\hat{\thetav})$ is given by the sandwich formula (\cite{Zeger1988}, equation (3.2))
\begin{equation}
Var(\hat{\thetav})=
\left[\sum_{j=1}^{n}\frac{\partial\mu'_j}{\partial\thetav} V_{j}^{-1}\frac{\partial\mu_j}{\partial\thetav}\right]^{-1}\left[\sum_{j=1}^{n}\frac{\partial\mu'_j}{\partial\thetav} V_{j}^{-1}Cov(Y_j)V_{j}^{-1}\frac{\partial\mu_j}{\partial\thetav}\right]\left[\sum_{j=1}^{n}\frac{\partial\mu'_j}{\partial\thetav} V_{j}^{-1}\frac{\partial\mu_j}{\partial\thetav}\right]^{-1}.
\end{equation}

For the crossover model (1), the $i$th element of $\frac{\partial\mu_j}{\partial\thetav}$  is $\frac{\partial\mu_{ij}}{\partial\thetav}=x'_{ij} \frac{\partial g^{-1}(\eta_{ij})}{\partial \eta_{ij}}$, where $x'_{ij}$ is the $i$th row of  $X_j$ for $i=1,\ldots,p$. The design matrix is $X_j =[1_{p} \;P_j \;T_j\;F_j]$, where
$P_j=I_p$; $T=(T'_1,\ldots,T'_n)'$, where $T_j$ is a $p\times t$ matrix with its $(i,s)$th entry equal to 1 if subject $j$ receives the direct effect of the treatment $s$ in the $i$th period and zero otherwise; $F=(F'_1,\ldots,F'_n)'$, where $F_j$ is a $p\times t$ matrix with its $(i,s)$th entry equal to 1 if subject $j$ receives the carryover effect of the treatment $s$ in the $i$th period and zero otherwise.

\subsection{Specific Case: Bernoulli Distribution}
If $Y_{ij}\sim \mbox{Bernoulli}(\mu_{ij})$, then the probability mass function of  $Y_{ij}$ is:
\begin{equation*}
f(y_{ij}) =  \mbox{exp}\left\{y_{ij}\mbox{log}\frac{\mu_{ij}}{1-\mu_{ij}}+\mbox{log}(1-\mu_{ij})\right\}
\end{equation*}

Comparing with equation (\ref{efd1}), we get $\phi_{ij} =  \mbox{log}\frac{\mu_{ij}}{1-\mu_{ij}}$, $b(\phi_{ij}) = -\mbox{log}(1-\mu_{ij}) = \mbox{log}(1+\mbox{exp}(\phi_{ij}))$, $c(y_{ij}) = 0$, $d(y_{ij}, \psi) = 0$ and $\psi = 1$. The mean of $Y_{ij}$ is $\mbox{E}(Y_{ij}) = \mu_{ij} = \frac{\mbox{exp}(\phi_{ij})}{1+\mbox{exp}(\phi_{ij})}$,
 and $\mbox{Var}(Y_{ij}) = \mu_{ij}(1-\mu_{ij})$.

Considering the logit link function to relate the linear predictor $\eta_{ij}$ to the mean $\mu_{ij}$, $g(\mu_{ij}) = \mbox{log}\frac{\mu_{ij}}{1-\mu_{ij}}$. Thus $g^{-1}(\eta_{ij}) = \frac{\mbox{e}^{\eta_{ij}}}{1+\mbox{e}^{\eta_{ij}}}$, and the $i$th component of $\frac{\partial\mu_j}{\partial\thetav}$  is $\frac{\partial\mu_{ij}}{\partial\thetav}=x'_{ij} \frac{\partial g^{-1}(\eta_{ij})}{\partial \eta_{ij}} = x'_{ij}\frac{\mbox{e}^{\eta_{ij}}}{(1+\mbox{e}^{\eta_{ij}})^{2}} = x'_{ij}\mu_{ij}(1-\mu_{ij})$.

\subsection{Specific Case: Poisson Distribution}
If $Y_{ij}\sim \mbox{Poisson}(\mu_{ij})$, then the probability mass function of $Y_{ij}$ is:
\begin{equation*}
f(y_{ij}) = \mbox{exp}\left\{y_{ij}\mbox{log}\mu_{ij} - \mu_{ij} - \mbox{log}y_{ij}!\right\}
\end{equation*}

Comparing with equation (\ref{efd1}), we get $\phi_{ij} = \mbox{log}(\mu_{ij})$, $b(\phi_{ij}) = \mu_{ij}$, $c(y_{ij}) = -\mbox{log}y_{ij}!$, $d(y_{ij}, \psi) = 0$ and $\psi = 1$. The mean of $Y_{ij}$ is $\mbox{E}(Y_{ij})$ = $\mu_{ij} = \mbox{e}^{\phi_{ij}}$ and  $\mbox{Var}(Y_{ij}) = \mbox{e}^{\phi_{ij}} = \mu_{ij}$.

Using the log link we obtain, $g(\mu_{ij}) = \mbox{log}(\mu_{ij}) = \eta_{ij}$, and the $i$th component of $\frac{\partial\mu_j}{\partial\thetav}$  is $\frac{\partial\mu_{ij}}{\partial\thetav}=x'_{ij} \frac{\partial g^{-1}(\eta_{ij})}{\partial \eta_{ij}} = x'_{ij}\mbox{e}^{\eta_{ij}} = x'_{ij}\mu_{ij}$.

\subsection{Specific Case: Gamma Distribution} \label{gamma}
If $Y_{ij}\sim \mbox{Gamma}(\kappa, \lambda_{ij})$, where $\kappa>0$ is the shape parameter (assume it is known) and $\lambda_{ij}>0$ is the rate parameter. Then the probability density function of $Y_{ij}$  is:
\begin{equation*}
f(y_{ij}) = \mbox{exp}\left\{\frac{y_{ij}\left(-\frac{\lambda_{ij}}{\kappa}\right) + \mbox{log}\lambda_{ij} + \mbox{log}y_{ij}}{1/\kappa} - \mbox{log}y_{ij} -\mbox{log}\Gamma \kappa\right\}
\end{equation*}
Comparing with equation (\ref{efd1}), we get $\phi_{ij} = -\frac{\lambda_{ij}}{\kappa}$, $b(\phi_{ij}) = -\mbox{log}(\lambda_{ij})$, $c(y_{ij}) = \mbox{log}y_{ij}$, $d(y_{ij}, \psi) = -\mbox{log}{y_{ij}}({\Gamma \kappa})$ and $\psi = 1/\kappa$. The mean of $Y_{ij}$ is  $\mbox{E}(Y_{ij}) =  \mu_{ij} = \kappa/\lambda_{ij}$ and $\mbox{Var}(Y_{ij}) = \frac{k}{\lambda_{ij}^{2}}$.

As in the case of Poisson distribution we use a log link function, $g(\mu_{ij}) = \mbox{log}(\mu_{ij}) = \eta_{ij}$.
The $i$th component of $\frac{\partial\mu_j}{\partial\thetav}$  is $\frac{\partial\mu_{ij}}{\partial\thetav}=x'_{ij} \frac{\partial g^{-1}(\eta_{ij})}{\partial \eta_{ij}} = x'_{ij}\mbox{e}^{\eta_{ij}} = x'_{ij}\mu_{ij}$.

\section{Approximate Designs}
For finding optimal crossover designs for the logistic model we use the approximate theory as in \cite{Laska1983} and \cite{Kushner1997, Kushner1998}.
Suppose ${\Omega}$ is the set of treatment sequences of the form $\omega=(t_1,\ldots,t_p)', \,t_i\in \{1,\ldots,t\}$, and
 $n_\omega$ is the number of subjects assigned to sequence $\omega$. Then, $n=\sum_{\omega\in\Omega}n_\omega,n_\omega\geq 0$. A design $\zeta$ in approximate theory is specified by the set $\{p_\omega,\omega\in\Omega\}$ where $p_\omega=n_\omega/n$, is the proportion of subjects assigned to treatment sequence $\omega$.

The matrices $T_j$ and $F_j$ depend only on the treatement sequence $\omega$ to which the $j$th subject is assigned,  so $T_j=T_\omega,\,F_j=F_\omega$, implying, $X_j=X_\omega$. Thus, the variance of $\hat{\thetav}$ is
\begin{equation}\label{optcrt}
Var_\zeta(\hat{\thetav})=\left[\sum_{\omega\in\Omega}np_\omega\frac{\partial\mu'_\omega}{\partial\thetav} V_{\omega}^{-1}\frac{\partial\mu_\omega}{\partial\thetav}\right]^{-1}\left[\sum_{\omega\in\Omega}np_\omega\frac{\partial\mu'_\omega}{\partial\thetav} V_{\omega}^{-1}Cov(Y_\omega)V_{\omega}^{-1}\frac{\partial\mu_\omega}{\partial\thetav}\right]\left[\sum_{\omega\in\Omega}np_\omega\frac{\partial\mu'_\omega}{\partial\thetav} V_{\omega}^{-1}\frac{\partial\mu_\omega}{\partial\thetav}\right]^{-1}.
\end{equation}
If the true correlation of $Y_j$ is equal to $R(\alpha)$ then we have a much simpler form, \begin{equation}
Var_\zeta(\hat{\thetav})=\left[\sum_{\omega\in\Omega}np_\omega\frac{\partial\mu'_\omega}{\partial\thetav} V_{\omega}^{-1}\frac{\partial\mu_\omega}{\partial\thetav}\right]^{-1}.
\end{equation}

\subsection{Design criterion}
In repeated measures trials when the interest is in only estimating direct treatment effect contrasts, we  may instead work with $Var(\hat{\tauv})$  given by,
\begin{equation}\label{pobj}
Var_\zeta(\hat{\tauv})=EVar_\zeta(\hat{\thetav})E',
\end{equation}
where $E$ is a $t\times m$ matrix given by $[0_{t1},0_{tp},I_t,0_{tt}]$ and $m$ is the total number of parameters in $\thetav$.

For obtaining D-optimal designs the function
\begin{equation}
\Lambda(\zeta,\thetav, \alpha) =\log\text{Det}(Var_\zeta(\hat{\tauv})).
\end{equation}
is minimized with respect to the design $\zeta$. Since it is a GLM the variance depends on the model parameters as well as the covariance parameters, and the design obtained is locally optimal.

To obtain D-optimal designs robust to uncertainties in the parameters we propose a Bayesian approach. This method has been used before for  logistic regression by \cite{Chaloner1989}, and \cite{Dror2006} and for block designs by \cite{Woods2011}. For repeated measures models, the design which minimizes
\begin{equation}\label{objectivefn}
 \Psi(\mathfrak{B},\zeta,\alpha) = \int_{\mathfrak{B}}\Lambda(\zeta,\thetav, \alpha)\,dF(\thetav),
\end{equation}
where $\mathfrak{B}\subset\mathbb{R}^{m}$ is the parameter space of  parameter vector $\thetav$ and $F(\thetav)$ is a proper prior distribution for $\thetav$, is the $D$-optimal Bayesian crossover design. Note, no  prior distributions are assigned to the correlation parameters $\alpha$, designs are obtained only for some fixed values chosen for $\alpha$.

In our computations we have used both uniform and normal priors for $\thetav$. The minimization of the objective function in  (\ref{objectivefn}) with respect to $\zeta$, requires high-dimensional integral calculation. In a GEE setup with blocks,  \cite{Woods2011} uses Latin Hypercube Sampling (LHS) for deriving an approximate solution of the above optimization problem.  For uniform priors, we use the average of (\ref{objectivefn}) across 100-point discrete samples using LHS as the approximate solution of (\ref{objectivefn}). When $\thetav$ has a Gaussian distribution, Latin Hypercube Sampling from Gaussian fields is used (for more details see \cite{Stein1987}).

For evaluating the performance of design $\zeta$ with respect to the reference design $\zeta^*$ (D-optimal Bayesian design), we use the  D-efficiency criterion defined as:
\begin{equation}\label{eff}
Eff_D(\zeta,\zeta^{*},\mathfrak{B},\alpha) = \left(\frac{\Psi(\mathfrak{B},\zeta^{*},\alpha)}{\Psi(\mathfrak{B},\zeta,\alpha)}\right)^{1/m},
\end{equation}
here $m$ is the number of model parameters.

Working correlation matrix structures such as the compound symmetric (or equi-correlated) and the AR(1) are investigated.
Under the equi-correlated covariance structure,
$R_j=(1-\alpha)I_p+\alpha J_p$,
and under the AR(1) assumption,
$
R_j=\alpha^{|i-i'|},\,i\neq i'.
$

\section{Examples}
\subsection{Example 1: Four periods, four treatments binary  response trial}
In Case study 1,  a four periods four treatments crossover trial described in \cite{Kenward1992} is considered. There are eighty subjects allocated to the four treatment sequences, with about twenty subjects per sequence. Treatments are denoted by $A$, $B$, $C$ and $D$. The treatment sequences form a \textit{Williams design} given as follows:

\begin{equation*}
 \begin{bmatrix}
A& B &C& D \\
 B&D&A&C\\
C&A&D&B\\
D&C&B&A
 \end{bmatrix}
\end{equation*}
The response variable is binary in nature. The data set is available in Table 3 of \cite{Kenward1992}.
For a four periods, four treatments trial, there are 24 possible \textit{Latin square designs} (LSDs)   with  every treatment  represented once and only once in each row and in each column (see Table 5.1 \cite{Senn_2002}). A special form of \textit{Latin square design} is called \textit{Williams square design} (WSD) in which every treatment follows every other treatment only once. For normal response when $t = p$ and $t$ is even, for reduced models LSD and  for full models WSD are  variance balanced designs (\cite{John2014}, page 361). However, these designs may not be optimal in general. But under some subject constraints WSD is universally optimal for even $t$, $n\leq t(t+2)/2$ and $4\leq t\leq12$ (\cite{Bose2009}, page 40).

The linear predictor $\eta_{ij}$ where $i$ stands for period and $j$ for subjects is modeled using several dummy variables as,
\begin{equation}
\eta_{ij} = \mu + \beta_{1}^{*}P_{1} + \beta_{2}^{*}P_{2} + \beta_{3}^{*}P_{3} + \tau_{1}^{*}T_{1} + \tau_{2}^{*}T_{2}+\tau_{3}^{*}T_{3} + \rho_{1}^{*}C_{1}+ \rho_{2}^{*}C_{2}+ \rho_{3}^{*}C_{3},
\end{equation}
where $P_{i}$ is the indicator variable for $i$th period, $T_{k}$ is the indicator variable for $k$th direct treatment effect and $C_{k}$  is the indicator variable for $k$th carryover effect.

Point estimates and corresponding confidence intervals of the parameters are calculated using \textit{PROC GENMOD} procedure in SAS software \cite{SAS1999}. Results are summarized in Table \ref{estimatebin} for both reduced and full models. In a reduced model it is assumed that there are no carryover treatment effect, while in a full model both direct and carryover treatment effects are assumed to be present. The working correlation structure is taken to be compound symmetric (CS) in nature,  the correlation coefficient is estimated to be 0.215.

\begin{table}[H]
\caption{Point estimates and confidence intervals for binary data in Example 1.}
\begin{center}
\begin{tabular}{c|cccccccccc}
Parameter & \multicolumn{2}{c}{ Point estimate [95\%Confidence interval]}\\
\hline
\mbox{}& no carryover effect & with carryover effect\\
\hline
$\mu$ & 1.0980 [0.4232\;1.7728] & 1.0158  [0.3474\; 1.6842]\\
$\beta_{1}^{*}$ & -0.3056 [-0.8643\;0.2532]&  -0.5525  [-1.2565\; 0.1515]\\
$\beta_{2}^{*}$ & -0.2414 [-0.8228 \;0.3399]& -0.4842  [-1.2034\; 0.2349]\\
$\beta_{3}^{*}$ &  0.3817 [-0.2391 \;1.0026]& 0.1234  [-0.6888\; 0.9356]\\
$\tau_{1}^{*}$  &-0.3270 [-0.8660\;0.2119]& -0.2564  [-0.8075\; 0.2948]\\
$\tau_{2}^{*}$ &  -0.0681 [-0.6996\;0.5635]&  0.0069  [-0.6473\; 0.6610]\\
$\tau_{3}^{*}$ &  -0.5322 [-1.1684\;0.1041]&  -0.3736  [-1.0165\; 0.2693]\\
$\rho_{1}^{*}$ &-& 0.1786  [-0.5965\; 0.9538]\\
 $\rho_{2}^{*}$ &-& 0.2242  [-0.5443\; 0.9927]\\
 $\rho_{3}^{*}$ &-& 0.6620  [-0.1352\;1.4591]\\
\hline
\end{tabular}\label{estimatebin}
\end{center}
\end{table}

Using the parameter estimates and confidence intervals in Table \ref{estimatebin}, we determine the Bayesian crossover design  by minimizing formula \ref{objectivefn}, and denote it by $D^{B}$. Instead of searching over all possible treatment sequences ($4^4$), the search space $\Omega$, is restricted  to 16 sequences \{$ACDB$, $BDCA$, $CBAD$, $DABC$, $ADCB$, $BCDA$, $CABD$, $DBAC$, $AABB$, $BBAA$, $CCDD$, $DDCC$, $AAAB$, $BBBA$, $CCCD$, $DDDC$\}. These sequences are so chosen that they can be used to form LSDs (including WSDs) and also non LSDs.

We use two types of priors for the regression parameters, the uniform and the normal. For the uniform prior, direct product of the confidence intervals of the parameter estimates in Table \ref{estimatebin} is used to represent the parameter space $\mathcal{B}$. While, for the two normal priors the parameter estimates are used in the mean vector and the correlation matrix is assumed to have an independent structure with variances 0.25 and 0.50, respectively. We use $\alpha=0.1,0.2,0.4,0.6,0.8$.

 The performance of $D^{B}$ is compared with  24 LSDs including 6 WSDs, and 24 extra period designs (EPDs) (a design in which first three rows correspond to a LSD and the last row is same as the previous one (\cite{Patterson1959}). We noted that the performance of each LSD is same among the 18 LSDs  under the reduced and full models for both of the correlation structures and priors used. Same is true for 6 WSDs and 24 EPDs. Thus the results are based on one LSD, one WSD and one EPD.

\subsubsection{Reduced model: No carryover effects}
Under the independent correlation structure ($\alpha$ = 0), $D^{B}$ utilizes all the 16 sequences with equal proportions. Under CS structure, for small values of $\alpha$,  70\% weightage is given to the sequences forming a LSD. As $\alpha$ increases, however, $D^B$ utilizes the sequences \{$ADCB$, $BCDA$, $CBAD$, $DABC$\} forming a LSD with almost 100\% weight and with equal proportions. Under the AR(1) structure, $D^B$ uses only the first eight sequences with equal proportions. From Figure \ref{effbinnc}, it is noted that under CS structure both LSD and WSD  are as good as $D^{B}$, while EPD performs badly. Efficiencies of LSD and WSD are constant with respect to $\alpha$. Under the AR(1) structure,  WSD is most efficient followed by LSD, and EPD performs worst. Note that performance of LSD and EPD designs  decrease as $\alpha$ increases.
\begin{center}
\begin{figure}[ht]
\begin{centering}
\includegraphics[scale=0.55]{p4t4_nc_cs_ar.eps}
\par\end{centering}
\vspace{-0.5cm}
\caption{Efficiency plots of designs computed for a uniform prior under the reduced model (a) exchangeable correlation structure (CS) and (b) AR(1) correlation structure for Example 1}
\label{effbinnc}
\end{figure}
\end{center}
\subsubsection{Full model: With carryover effect}
For $\alpha = 0$, $D^{B}$ utilizes the first eight sequences with equal proportions and more than 80\% weight. Under the CS structure, $D^B$ utilizes the first eight sequences with more than 70\% of weight. As $\alpha$ increases the first eight sequences get more than 90\% weight with equal proportions. For the AR(1) structure,  $D^B$ again utilizes the first eight sequences with more than 70\% of weight and the results do not change with $\alpha$. It can be observed from Figure \ref{effbinwc} that WSD is most efficient compared to LSD and EPD under both correlation structures. Efficiency of WSD reduces marginally with an increase in $\alpha$, while efficiencies of the other designs decrease more as compared to WSD with $\alpha$. Results are same when normal priors used.

\begin{center}
\begin{figure}[ht]
\begin{centering}
\includegraphics[scale=0.55]{p4t4_ar_cs_wc.eps}
\par\end{centering}
\vspace{-0.5cm}
\caption{Efficiency plots of designs computed for a uniform prior under the full model (a) exchangeable correlation structure (CS) and (b) AR(1) correlation structure for Example 1}
\label{effbinwc}
\end{figure}
\end{center}

\subsection{Example 2: Two periods two treatments Poisson  response trial}
A crossover trial with two drugs given in two periods for controlling angina in 20 patients is considered. The count of attacks suffered by the patients is assumed to be a Poisson variable.  Treatment sequences  $AB$ and $BA$ are used in the trial. However, we should note that this design does not permit the unbiased estimation of the treatment contrast under carryover effect (\cite{Jones_2014}),  though the estimates and corresponding confidence intervals may still be used to choose the prior distributions.

The linear predictor $\eta_{ij}$ for this $2\times 2$ crossover design is,
\begin{equation*}
\eta_{ij} = \mu + \beta P +  \tau T+ \rho C,
\end{equation*}
where $P, T$ and $C$ are the indicator variables for the period, direct treatment and crossover effects, respectively.
 For a $2\times2$ cross-over trial  compound symmetric and AR(1)  correlation structures are equal. Estimation
of the parameters is again done by using \textit{PROC GENMOD} in SAS software \cite{SAS1999}. Point estimates and their 95\% confidence intervals are listed in Table \ref{estimateP}. Estimate of the correlation coefficient is $\alpha = 0.0798$.
\begin{table}[H]
\caption{Point estimates and confidence intervals for Poisson data in Example 2.}
\begin{center}
\begin{tabular}{c|cccccccccc}
Parameter & \multicolumn{2}{c}{ Point estimate [95\%Confidence interval]}\\
\hline
\mbox{}& no carryover effect & with carryover effect\\
\hline
$\mu$ & 0.0493 [-0.4457 \;0.5444]&-0.0541 [-1.0405\; 0.9324] \\
$\beta$ & -0.0011 [-0.4256 \;0.4234]&0.0541 [-0.4519\; 0.5600]\\
$\tau$ &  0.5664 [0.1006\;1.0322]&0.6419 [-0.1036\; 1.3873]\\
$\rho$ &-&0.1494 [-0.8566\; 1.1553]\\
\hline
\end{tabular}\label{estimateP}
\end{center}
\end{table}

For finding the Bayesian crossover designs we search over the space of all treatment sequences, \{$AB,BA,AA,BB$\}. The Bayesian design with the $D$-optimal allocation of subjects to the treatment sequences \{$AB,BA,AA,BB$\} is denoted by $D^B$.  The performance of $D^B$  is compared to  $D_{I}=\{AB,BA\}$ with equal allocation to each sequence, and another design $D_{II}$ with  treatment sequences \{$AB,BA,AA,BB$\}, but   with equal allocation to each treatment sequence.
The parameter space $\mathcal{B}$ is similarly chosen for both uniform and normal priors as in Example 1. The results of the design comparisons is given as efficiency plots in Figure \ref{efft2p2}.

\subsubsection{Reduced model: No carryover effects}

 For all values of $\alpha$, the Bayesian crossover design $D^B$ for the reduced model consists of sequences \{$AB$, $BA$\} with approximately equal proportion of subjects to each sequence, thus $D_{I}$ and $D^B$ are similar under the reduced model. However, the performance of $D_{II}$ is affected by increasing values of the correlation parameter. The results matches with that of the normal response model for a $2\times 2$ crossover design (\cite{Laska1985}).

\subsubsection{Full model: With carryover effects}

 Introducing crossover effects in the model, however changes the results completely. The Bayesian crossover design $D^B$ for full model utilizes the sequences \{$AA$, $AB$\} and its dual with approximate equal proportion of subjects to each sequence, thus $D_{II}$ and $D^B$ are similar under the full model, for all $\alpha$. The original design $D_{I}$ considered in the trial has very low efficiency compare to $D^B$ and is also affected by increasing $\alpha$. This result again matches that of  a normal response with carryover model, where treatment sequences \{$AA$, $AB$, $BB$, $BA$\} with equal proportions of the subjects are optimal  for estimating treatment contrasts (\cite{Laska1985}).

\begin{center}
\begin{figure}[ht]
\includegraphics[scale=0.55]{poisson_uni_normal_eff.eps}
\caption{Efficiency plots of designs computed under a uniform prior for (a) reduced model (b) full model for Example 2}
\label{efft2p2}
\end{figure}
\end{center}

\subsection{Example 3: Three periods two treatment Gamma response trial}
We consider a hypothetical gamma response trial with two treatments, $A$ and $B$ applied in three periods. The response is length of hospital stay which is assumed to follow a Gamma distribution. The Bayesian crossover design $D^{B}$ is determined by searching over the treatment sequences, $AAB, ABB, ABA, BBA, BAA, BAB$. Linear predictor for this crossover design is
\begin{equation*}
  \eta_{ij} = \mu + \beta_{1}P_{1} + \beta_{2}P_{2} + \tau T + \rho C
\end{equation*}
where $P_{i}$ is again the indicator variable for $i$th period, $T$ is the indicator variable for treatment and $C$ is the indicator variable for carryover effect.

The data sets are simulated using the parameter values $(\mu, \beta_{1},\beta_{2}, \tau) = (0.50, 0.15, 0.20, 0.25)$ for a reduced model and $(\mu, \beta_{1},\beta_{2}, \tau, \rho) = (0.50, 0.20, 0.30, 0.25, 0.15)$ for a full model. We have considered the treatment sequences $ABB$ and $BAA$ with the assignment of 10 subjects each to generate the data. Observations are assumed to be independent within the periods. The link function used is defined in Section \ref{gamma} and the shape parameter $\kappa$ is fixed at 2.0. The point estimates and the confidence intervals listed in Table 5 are used to create the parameter space, $\mathcal{B}$, for both the normal and uniform priors.
We compared the Bayesian $D$-optimal design $D^{B}$ with the following designs:
\begin{itemize}
\item[$D_{a}$] = $ABB, BAA, AAB, BBA$ with equal allocation to each treatment sequence.
\item[$D_{b}$] = $ABB, BAA$ with equal allocation to each treatment sequence.
\item[$D_{c}$] = $ABA, BAB, ABB, BAA$ with equal allocation to each treatment sequence.
\end{itemize}

\begin{table}[H]
\caption{Point estimates and confidence intervals for Gamma response in Example 3.}
\begin{center}
\begin{tabular}{c|cccccccccc}
Parameter & \multicolumn{2}{c}{Point estimate [95\% Confidence interval]}\\
\hline
\mbox{}& no carryover effect & with carryover effect\\
\hline
$\mu$ &  0.5455 [0.3590\;0.7321]&  0.3674 [0.2135\; 0.5213] \\
$\beta_{1}$ & 0.1567 [0.0433\;0.3567] &  0.2271 [-0.0547\;  0.5089]\\
$\beta_{2}$ & 0.14 [-0.2324\; 0.2592] &  0.2498 [0.0245\; 0.4751]\\
$\tau$ &   0.1724 [0.0065\;0.338] &  0.1134 [-0.1335\; 0.3603]\\
$\rho$ &-&  0.1477 [-0.1621\; 0.4575]\\
\hline
\end{tabular}\label{estimateP}
\end{center}
\end{table}

\subsubsection{Reduced model: No carryover effects}
First we consider the uniform prior distribution for the parameters.
Under independent correlation structure ($\alpha$ = 0), Bayesian $D$-optimal design $D^B$ utilizes all the sequences $\{AAB, ABB, ABA, BBA, BAA, BAB\}$ with the proportions of (21\%, 12\%, 21\%, 13\%, 21\%, 13\%) to each sequence. From Figure \ref{effgammanocarry} for $\alpha = 0$, it is noted that designs $D_{a}$, $D_{b}$ and $D_{c}$ are as efficient as $D^B$ with efficiency approximately equal to 1.

Under the CS structure, $D^B$ uses \{$AAB$, $ABA$, $BBA$, $BAA$ \} sequences with proportions (46\%, 8\%, 38\%, 8\%). As correlation increases (near $\alpha = 0.8$), $D^B$ assigns almost (50\%) proportions to each of the sequences \{$AAB$, $BBA$\}. Observing Figure \ref{effgammanocarry} (a), design $D_{a}$ turns out to be more efficient compared to $D_{b}$ and $D_{c}$. Performance of all designs decrease as $\alpha$ increases also, designs $D_{b}$ and $D_{c}$ are quite similar in their efficiencies.

For the case of AR(1) structure, $D^B$ utilizes the sequence $ABA$ and its dual with equal proportions for all values of $\alpha$. Design $D_{c}$ is  more efficient compared to $D_{a}$ and $D_{b}$ while $D_{b}$ performs the worst.  Results stay same for normal priors.

\subsubsection{Full model: With carryover effects}

Under independence correlation structure, $D^B$ utilizes the sequences \{$ABB$, $AAB$\} and their dual with approximately equal proportions. When a carryover effect is included in the model  $D^B$ utilizes the sequence $ABB$, $AAB$ and their dual with approximately equal proportions for each  $\alpha$ under CS correlation structure. Thus, $D^B$ is similar to $D_{a}$. Under the AR(1) correlation structure, $D^B$ utilizes the sequence $ABB$, $AAB$ and their dual with approximately equal proportions for $\alpha<0.2$, for $0.2\leq\alpha\leq0.5$, it gives about 70\% weight to \{$AAB$, $BBA$\}, $28\%$ to \{$ABB$, $BAA$\} and includes $BAB$ in the design. For higher values of $\alpha$, $D^B$ utilizes $AAB$ and $BBA$ with more than 80\% weight  and the rest goes to $ABB$ and $BAB$.

From Figure \ref{effgammawc} we see that the designs $D_{b}$ and $D_{c}$ perform badly when compared to $D_a$ (similar to $D^B$) for all values of $\alpha$ and for both CS and AR(1) structures.
The results remain unchanged to changes in the scale parameter value $\kappa$. Use of normal prior provides the same results. Under AR(1) correlation structures, \cite{Laska1985} reported similar observations  for normal responses.

\begin{center}
\begin{figure}[ht]
\includegraphics[scale=0.5]{eff_cs_ar_unif_gamma.eps}
\caption{Efficiency plots of designs computed for a uniform prior for  reduced model under (a) exchangeable correlation structure (CS) and (b) AR(1) correlation structure for Example 3}
\label{effgammanocarry}
\end{figure}
\end{center}

\begin{center}
\begin{figure}[ht]
\includegraphics[scale=0.62]{eff_cs_ar_unif_gamma_wc.eps}
\caption{Efficiency plots of designs computed for a uniform prior for full  model under (a) exchangeable correlation structure (CS) and (b) AR(1) correlation structure for Example 3}
\label{effgammawc}
\end{figure}
\end{center}
\section{Concluding Remarks}
Crossover designs are popular as designs of choice in many clinical
and pharmaceutical trials for comparing treatments.  However, very often  in these situations the response does not follow the usual assumptions of normality, and  generalized linear models have to be used to model the data.
In this article, we address the designing of such crossover trials when a GLM is fitted. Since the designs are dependent on the model parameters, Bayesian designs are proposed. Comparing our main results based on GLMs with those of normal response models, we see that they are quite similar in many cases.

The main results on
the estimation of direct effects using the proposed D-optimal Bayesian designs  ($D^B$) are summarized below.
\begin{itemize}
\item{\textbf{For $t=p=4$ when the response is binary :}} Williams design is as efficient as $D^B$ and is seen to perform the best under both CS and AR(1) correlation structures for a reduced as well as a full model.
\item {\textbf{For $p=t=2$ when the response is Poisson distributed:}}
 Design \{$AB, BA$\} has the highest efficiency in a reduced model framework while for a full model, design \{$AB, BA, AA, BB$\} is most efficient. Both designs have equivalent efficiency as $D^B$ for the respective models.
\item {\textbf{For $p=3, t=2$ when the response is Gamma distributed:}}
For a reduced model, design \{$AAB$, $ABA$, $BBA$, $BAA$ \} is the $D$-optimal Bayesian design under CS structure for correlation $\alpha<0.6$. While for higher values of $\alpha$, design \{$AAB$, $BBA$ \} is the $D$-optimal Bayesian design.
For AR(1) correlation structure, design \{$ABA, BAB$\} turn out to be the $D$-optimal Bayesian design.
While for a full model, under CS structure, design \{$ABB, BAA, AAB, BBA$\} is the $D$-optimal Bayesian design. Under AR(1) correlation structure, $D^B$ is same as in case of CS structure for $\alpha<0.2$, but for $0.2\leq\alpha\leq0.5$, it gives more proportions (70\%) to $AAB$ and $BBA$. For higher values of $\alpha$, $D^B$ utilizes $AAB$ and $BBA$ with more than 80\% weight  and rest to \{$ABB$, $BAB$\}.
\end{itemize}
In many biological experiments while studying the effect of drugs, the response measured may not be binary in nature but say ordinal. As  an example consider a  $3\times 3$ crossover trial  (cited by  \cite{Jones_2014}) where the effect of three treatments on the amount of patient relief is studied. The response obtained is categorized as none, moderate or complete, making it  ordinal in nature with three categories. Thus, there is a need to address optimal crossover deigns not just for binary models but also for multi categorical responses. In these cases, instead of the logit link, a generalized logit or a proportional odds model may be  used. Also, other than the correlation between measurements from the same subject we would have to consider the relation between response categories.  \cite{Jones_2014} discusses modeling of ordinal data using the GEE approach. In future, we are interested to study D-optimal Bayesian designs for such multicategorical models.

\textbf{Acknowledgement:} Mr. Satya Prakash Singh wishes to thank University Grant Commission (UGC), India, for the award of a research fellowship. The work of Siuli Mukhopadhyay
was supported by the Department of Science and Technology, Government of India [Grant
Number: SR/FTP/MS-13/2009]. These supports are gratefully acknowledged.
\bibliographystyle{elsarticle-harv}


text:
\section{Introduction}
Given an static image, humans can think of various scenes of what will happen next using their imagination.  For example, considering the ballerina in Figure \ref{fig:dance}, one can easily picture the scene of the dancer jumping higher or landing softly. In this work, we clarify the task as intimating human capability of \textit{\textbf{Video Imagination}}:
synthesizing imaginary videos from single static image.
This requires synthesized videos to be diverse and plausible.
Although this study is still in its infancy, we believe video prediction and image reconstruction area can draw inspiration from it.

Compared to related tasks, e.g. video anticipation and prediction, there are more challenges for video imagination . Video imagination means to produce real high-dimension pixel values unlike low-dimension vectors in semantic anticipation. In addition, videos that are not identity to each other can all be reasonable, like imaginary video 1 and imaginary video 2 in Figure \ref{fig:dance}. So there is no precise ground truth as in common video prediction task. This intrinsic ambiguity makes regular criterion like MSE fails in evaluating whether the synthesized video is plausible. Moreover, compared to image generation, video synthesis needs to additionally model the temporal dependency that makes consecutive frames seem realistic.

Pioneers make attempts. Dense trajectory \cite{walker2016uncertain} and optical flow \cite{pintea2014deja} have been used to model scene dynamics. Variational auto-encoder \cite{walker2016uncertain} and stochastic Markov-chain \cite{ranzato2014video} have been introduced to form generative model. However, those models still struggle in high dimension space where manifold is hardly tractable, and is unsatisfying in terms of criterion
\begin{figure}
	\includegraphics[width=3in]{dance}
	\caption{Synthesizing multiple imaginary videos from one single image. \textrm{For instance, given an image of a dancing ballerina, the videos of the dancer jumping higher or landing softly are both plausible imaginary videos. Those videos can be synthesized through applying a sequence of transformations to the original image}.}
	\label{fig:dance}
\end{figure}

In this work, we present an end-to-end unsupervised framework with transformation generation for video imagination .
Our key intuition is that we can model in transformation space instead of pixel space.
Since scenes in frames are usually consistent, we assume that the major motions between frames can be modeled by transformations.
If we reconstruct frame based on the original image and corresponding transformations, both scene dynamic and invariant appearance can be preserved well.
In addition, we draw inspiration from image generation works \cite{radford2015unsupervised} that use adversarial training. We believe an elaborate critic network that understands both spatial and temporal dependency would serve as reasonable criterion.

Based on the intuition and inspiration above, we design our framework focusing on model distributions in transformation space implicitly, and train it in adversarial way.
In this framework, we generate transformation conditioned on the given image. Then we reconstruct each frame by applying the generated transformation to the given image. Latent variable is also introduced to enable diverse sampling. Casting this into an adversarial architecture,  we train our framework in a fully end-to-end fashion.

We believe this framework is a promising way to overcome existing challenges. As we build generation model in transformation space, it is more tractable to implicitly model the distribution of transformation. Conditioned on image makes generated transformation reasonable. The procedure of applying transformation to original image is similar to the insight of highway connection \cite{he2016identity}, and this helps the synthesized video maintaining sharp and clear.
Also, the latent variable
enables diverse imagination through sampling different transformations corresponding to different imaginary videos.
Furthermore, there is nearly infinite resource for this unsupervised training. No label is needed, so every video clip can serve as a training sample.

For evaluation, since there is no general evaluation metrics for this task, we employ image quality assessment method to evaluate the quality of reconstructed frames and present a relative image quality assessment ($RIQA$) to eliminate the scene difference. In experiments, we evaluate our idea on three datasets, including two artificial video datasets with simple motions and one natural scene video dataset with complex motions.
The synthesized 4-frames video results show that our framework can produce diverse sharp videos with plausible motions.
We compare our framework with some related methods and two custom baselines. The quantitative evaluation results suggest that our framework outperforms others including those methods that are given more prior information, and the qualitative comparison also shows the advance of our synthesized videos.

The primary contribution of this paper is developing a new end-to-end unsupervised framework to synthesize imaginary videos from single image. We also make brave attempt on new evaluation method. In section 2, we review related work. In section 3, we present our \textit{Video Imagination} video synthesis framework in details. In section 4, we illustrate new evaluation method $RIQA$ and show experiments and comparison.

\begin{figure*}[h]
	\includegraphics[width=7in]{model}
	\caption{Pipeline of video imagination from single image. \textrm{In our framework, to produce one imaginary video, the input image is first encoded into a condition code and sent to transformation generator together with a latent variable. The generated transformation sequence is applied to input image later in volumetric merge network where frames are reconstructed with transformed images and volumetric kernels. Those four frames form one imaginary video. By sampling different latent variable from guassian distribution, our framework can produce diverse imaginary videos.} }
	\label{fig:model}
\end{figure*}

\section{Related work}

Although the works of future video synthesis from single image are rather little, our task shares common techniques with two related tasks: video prediction \cite{sutskever2014sequence} and image reconstruction \cite{vincent2008extracting}, where researchers have made impressive progress. In the following, we regard them as a universal visual prediction task, and review related works from different perspectives of approaches.

\textbf{Reconstruction in pixel space.} Early works of visual prediction focus on modeling and estimation in pixel space \cite{wang2004image} \cite{tan2006intra} \cite{wang2009mean}. These methods reconstruct images by calculating pixel values directly. With recent resurgence of deep networks, researchers tend to replace standard machine learning models with deep networks. In particular, \cite{kalchbrenner2016video} proposes a video pixel network and estimates the discrete joint distribution of the raw pixel values. \cite{srivastava2015unsupervised} uses LSTM network to learn representations of video and predict future frames from it. \cite{vondrick2016generating} employs adversarial training and generates video from scratch with deconvolution method \cite{zeiler2010deconvolutional}.
A key issue in pixel-level prediction is the criterion metrics. A recent work \cite{mathieu2015deep} argues that standard mean squared error (MSE) criterion may fail with the inherently blurry predictions. They replace MSE in pixel space with a MSE on image gradients, leveraging prior domain knowledge, and further improves using a multi-scale architecture with adversarial training.

\textbf{Mid-level tracking and matching.} To overcome the challenge of high dimensionality and ambiguity in pixel space, the prediction framework of mid-level elements gradually becomes popular. \cite{mahajan2009moving} explores a variation on optical flow that computes paths in the source images and copies pixel gradients along them to the interpolated images.  \cite{walker2014patch} combines the effectiveness of mid-level visual elements with temporal modeling for video prediction.\cite{ranzato2014video} defines a recurrent network architecture inspired from language modeling, predicting the frames in a discrete space of patch clusters. The input in \cite{walker2016uncertain} is a single image just like us, where the authors predict the dense trajectory of pixels in a scene with conditional variational autoencoder.

\textbf{Existing pixels utilization.} A insightful idea of improving the quality of prediction image is to utilize existing pixels. \cite{liu2017video} synthesizes video frames by flowing pixel values from existing ones through voxel flow. \cite{xue2016visual} outputs the difference image, and produces the future frame by sum up the difference image and raw frame. \cite{de2016dynamic} and \cite{finn2016unsupervised} share a similar methods with us of applying filters to raw frames to predict new frames, and they provide the validation of gradients flow through filters.

\textbf{Generation model evolutions.} Traditional works treat visual prediction as a regression problem. They often formulate prediction tasks with machine learning techniques to optimize the correspondence estimation \cite{kitani2012activity,lan2014hierarchical,hoai2014max}. With the development of deep networks, community of visual prediction has begun to produce impressive results by training variants of neural network structures to produce novel images and videos \cite{xie2016synthesizing,gregor2015draw,xie2016deep3d,zhou2016view}. The probabilistic models become popular again. More recently, generative adversarial networks (GANs) \cite{goodfellow2014generative} and variational autoencoders \cite{kingma2013auto} have been used to model and sample from distributions of natural images and videos  \cite{radford2015unsupervised,denton2015deep,yan2016attribute2image}. Our proposed algorithm is based on GAN, but unlike previous works starting with a simple noise, we force our generation model conditioned on the given image, which benefits to generate reasonable transformation.

To the best of our knowledge there are no existing model that can produce multiple videos given one single image. Perhaps the most similar works to our task are \cite{xue2016visual,walker2016uncertain}, where both works aim to build a probabilistic model of future given an image. But \cite{xue2016visual} only outputs one frame and \cite{walker2016uncertain} just produce optical flows.

Also, note a concurrent work that learns to predict in transformation space is \cite{van2017transformation}, where the authors predict the new frames by predicting the following affine transformations. But their task is to generate frames from sequence of frames while ours is to synthesize imaginary videos given a single image. In addition, our work differs in that there methods are close to a regression problem as to predict precise future frames, but our task requires a probabilistic view and aims at generating multiple videos.

\section{Approach}

Rather than struggle in high-dimension pixel space, our idea is to model in transformation space for video imagination: to take one single image as input and synthesize imaginary videos that picture multiple plausible scene change of that image. Figure \ref{fig:model} shows the pipeline of output one imaginary video in our framework.

In our framework, firstly, we send latent variable and condition code encoded from image into \textbf{transformation generator}, which outputs a group of transformation sequences. Secondly, we apply those transformation sequences to the original image and reconstruct frames through a \textbf{volumetric merge network}. Finally, we combine frames as an imaginary video then use \textbf{video critic network} to achieve adversarial training.
In the following subsections, we firstly give a problem description; then we describe the details of those crucial parts and its implementation.

\subsection{Problem definition  }

Firstly we use formulations to describe this task: given an image $X$, outputs $m$ imaginary videos $\hat V$
corresponding to different reasonable motions. Each imaginary video contains $T$ consecutive frames $f_T$.

Ideally, we would like to model the distribution $P(V \mid X)$ of all possible imaginary $V$ given $X$. Practically, we aim to train a neural network $T_{\theta}$ with parameters $\theta$, which implicitly models a distribution $P_T(V \mid X)$. Through training, we expect $P_T(V \mid X)$ to converge to a good estimate of $P(V \mid X)$. $T_{\theta}(X)$ yields a sample $\hat V$ drawn from $P_T(V \mid X)$, so we have
\begin{equation}
\hat V = T_{\theta}(X) \sim P_T(V \mid X)
\end{equation}

Instead of directly modeling in pixel space, we choose to model distribution in transformation space. We build this model based on a key assumption that the major motions between frames can be modeled by transformations. That means letting $M_T$ denote motion between $X$ and $f_T$, $M_T$ can be represented by a transformation sequence $\Phi_T$ containing $p$ transformations. Letting $\odot$ denote the operation of apply transformation sequence to image, we have $ f_T = \Phi_T \odot X$. Letting $\Phi$ represent the group of transformation sequences of all videos, we have $V = \Phi \odot X$. By introducing $G$ implicitly modeling $P_G(\Phi \mid X)$ in transformation space, we have a new description of target:
\begin{equation}
\hat V = G_{\theta}(X) \odot X  \sim P_T(\Phi \odot X \mid X) = P_G(\Phi \mid X) \odot X
\end{equation}

To make diversity samplings of $V$ feasible, we introduce latent variable $z$ that follows a specific distribution (e.g. Guassian Distribution). Hence, we can modify target of $G_\theta$ from  modeling the distribution $P_G(\Phi \mid X)$ to modeling $P_G(\Phi \mid X, z)$.
This implicit distribution allow us to sample different imaginary videos $\hat V$ through sampling different $z$. Therefore, everything reduces to the following target:
\begin{equation}
\hat V = G_{\theta}(X, z) \odot X \sim P_G(\Phi  \mid X, z)\odot X
\end{equation}

\subsection{Transformation Generator}
The job of transformation generator is implicitly modeling $P_G(\Phi \mid X, z)$ so that it can generate transformation conditioned on image. Given the condition code of a static image $X$, together with a latent variable $z$, the goal of transformation generator is learning to generate a transformation group $\Phi$.

To be specific, transformation generator outputs $T$ transformation sequences $\{\Phi_1, \Phi_2, \cdots, \Phi_T\}$ corresponding to transformations between $X$ and $\{f_1, f_2, \cdots,  f_T\}$. Each transformation sequence $\Phi_T$ contains $P$ transformations formed by $K$ parameters. Transformations are generated in a sequential fashion in hope of a better description of warp motion, because motion can often be decomposed in a layer-wise manner.

We use two kinds of transformations to model motion. Since in adversarial training, the gradient back-propagation starts from critic network then flows to the frames, the transformation type we choose needs to allow gradient propagating from transformed images to transformation generator. Fortunately, prior works in \cite{jaderberg2015spatial,de2016dynamic} revealed that there is a group of transformations having this adorable attribution. We build two distinct models to form $\Phi$ based on prior works.

\textbf{Affine Transformation: }Simply formed by 6 parameters, affine transformation can model motions including translation, rotation, zoom, and shear. Works in \cite{wang1993layered,bergen1990computing} have shown that affine transformation provides a good approximation of 3-D moving objects motion. Affine transformation works on coordinates, which would raise a problem of undefined pixel locations. In practice, we use differentiable bilinear interpolation to complete those pixels.

\textbf{Convolutional Transformation: }A convolution kernel can naturally model simple motions like translation, zoom and warp as shown in Figure \ref{fig:filter}. The kernel size can vary with application scene. For example, a $5 \times 5$ kernel allows pixels translating over a distance of 2 pixels. A sequence of kernel would raise the size of receptive field and allow more complex or intenser motions.
\begin{figure}
	\includegraphics[width=3in]{filter}
	\caption{Different convolution kernels result in different motions. \textrm{The doted square denotes convolution kernel and the right side image shows the result of applying the kernel. One simple kernel can model motion like b) translation c) Zoom d) Warp.}}
	\label{fig:filter}
\end{figure}

\subsection{Volumetric Merge Network }
Volumetric merge network is responsible for reconstructing frames $\{f_1, f_2, \cdots,  f_T\}$ based on the generated transformation $\Phi$ and image $X$. The transformation group $\Phi$ is finally applied to image $X$, producing an intermediate image group $I$ consisting of $T$ intermediate image sequences $\{{I_1}, {I_2},$ $ \cdots, {I_T}\}$ that will be used to reconstruct $\{f_1, f_2, \cdots,  f_T\}$ accordingly. Combining frames temporally, volumetric merge network outputs imaginary video $\hat V$.

Since the transformation is generated in a sequential fashion, it is intuitive to take the sequence of intermediate images as an extended dimension representing transformation.
That is, we consider each transformed sequence ${I_T}$ as one entity ${I_T} \in \mathbb{R}^{W \times H \times P}$ that has 3 dimensions as width $W$, height $H$, and transformations $P$.
This 3-D entity, as shown in Figure \ref{fig:volum}, allows us to reconstruct frame by merging it in a volumetric way. Each pixel is reconstructed through volumetric kernels. The kernels can take both neighbor pixel values and intermediate image differences into consideration.

Parameters in volumetric kernels can be obtained either from clipping a crop of generated transformations or through a specific volumetric kernel generator as shown in Figure \ref{fig:model}. Volumetric kernel generator (a full convolution network) concentrates more on capturing the dependency in spatial domain, while generated transformations can give volumetric kernel better understanding of correlation  between intermediate images.
\begin{figure}
\includegraphics[width=3in]{volum}
\caption{Intermediate image sequence $I_T$ as 3-D entity. \textrm{A volumetric kernel can take both neighbor pixel values and intermediate image differences into consideration.}}
\label{fig:volum}
\end{figure}

\subsection{Video Critic Network}
To meet the requirement of a better criterion, we design a video critic network $Critic$ to achieve adversarial training. Video critic network $Critic$ receives synthesized video $\hat V$ and real video $V$ as input alternatively, and outputs criticism judging how convincing the input is.

A convincing video means that the frame looks clear and the motion between frames seems consecutive and reasonable.
Video critic network $Critic$ needs to give reference of whether the input is plausible and realistic, which requires understanding of both static appearance and dynamic scene. The similar requirement can be found in action recognition task, where lately researchers have made progress \cite{tran2015learning}. We draw inspiration from those works, and design $Critic$ to have the structure of  spatial-temporal convolution networks \cite{ji20133d}.

\begin{figure*}[h]
	\includegraphics[width=7in]{r6in1}
	\caption{Quality Performance of our framework. \textrm{In each dotted box, the first shows the synthesized imaginary videos given the fist frame as input. The second row shows the difference images of synthesized frames and input. (a)(b) demonstrate the results experiment on moving MNIST and 2D shapes dataset. (c)(d) shows the result of surfing class on UCF101 dataset in different resolution as c) $64 \times 64$ and d) $128 \times 128$. (e)(f) shows the results given image  from swing and ice-dancing categories in UCF101 dataset. The synthesized frames are sharp and clear. Difference images illustrate plausible motions. Results of different resolutions and different image categories on UCF101 dataset suggest our framework shows scale to the complexity of high-resolution videos. }}
	\label{fig:result4in1}
\end{figure*}

\subsection{Learning and Implementations}
Our framework consists of fully feed-forward networks.
The transformation generator consists of 4 fully connected layers. The latent code sampled from a guassian distribution has 100 dimensions, and the condition code has 512 dimensions. We can encode $X$ into condition code either through refined AlexNet or a 5 layer convolutional network.
The volumetric merge network consists of 3 volumetric convolutional layers, while the last layer uses element-wise kernel.
We use a five-layer spatio-temporal convolutional network as the critic network.

We employ Wasserstein GAN \cite{arjovsky2017wasserstein} to train our framework. The generator loss $L_g$ is defined as:
\begin{equation}
	L_g = - \mathbb{E}_{v \sim P_T(V \mid X)}  Critic(v)
\end{equation}
The critic loss $L_c$ is defined as:
\begin{equation}
L_c = \mathbb{E}_{v \sim P_T(V \mid X)}  C(v) - \mathbb{E}_{v \sim P(V \mid X)} Critic(v)
\end{equation}

Alternatively, we minimize the loss $L_g$ once after minimizing the loss $L_d$ 5 times until a fixed number of iterations.
Ultimately, the optimal video critic network $C$ is hoped to produce good estimate of Earth-Mover (EM) distance between $P(V \mid X)$ and $P_T(V \mid X)$.
We use the RMSProp optimizer and a fixed learning rate of 0.00005. ReLU activation functions and batch normalization are also employed.

We use a Tesla K80 GPU and implement the framework in TensorFlow \cite{abadi2016tensorflow}. Our implementation is based on a modified version of \cite{radford2015unsupervised}, and the code can be found at the project page\footnote{https://github.com/gitpub327/VideoImagination} \footnote{ This page contains no information about the authors}. Since we model in relatively small transformation space, the model converges faster than others. Training procedure typically takes only a few days even hours depending on datasets.

\section{Experiment}

In this section, we experiment our framework on 3 video datasets: Moving MNIST \cite{srivastava2015unsupervised}, 2D shape \cite{xue2016visual} and UCF101 \cite{soomro2012ucf101}. For evaluations, we perform qualitative inspection and novel quantitative assessment $RIQA$ to measure the objective quality of the imaginary video.
\subsection {Baselines and Competing Methods}

Current work about this task is quiet limited. To find out whether our framework outperforms those methods that do not involve our crucial components, we develop two simple but reasonable baselines for this task. For the first one, \textbf{Baseline 1}, the transformation generator and volumetric merge network in our original framework are replaced by a generator network that directly outputs flatten pixels. For the second one, \textbf{Baseline 2}, the whole adversarial training procedure including critic network is removed, and the network is trained minimizing $l_2$ loss function. Those two baselines can also be considered as a form of ablation experiments.

We also consider several latest works as competing methods as shown in \ref{tab:task}. The task setting is distinct, so it is difficult to find evaluation metrics that can fairly compare all those works together, but we make brave attempt later in Section \ref{sub:eva} to compare our framework against some of those works.

\begin{table}[t]
	\caption{Task setting comparison of related work. \textrm{Multiple output means that the method build a probabilistic model and can sample different results. $\ast$ indicates the methods can also experiment on natural scenes like in UCF101 dataset.}}
	\label{tab:task}
	\begin{tabular}{ccc}
		\toprule
		Model&Input&Output\\
		\midrule
		Ours $\ast$ & image & 5 frames(multiple) \\
		Visual Dynamic \cite{de2016dynamic} & image& 1 frame(multiple)\\

		Scene Dynamic \cite{vondrick2016generating} & image& 32 frames \\
		Dynamic Filter \cite{de2016dynamic} & 4 frames&1 frame \\
		Beyond MSE \cite{mathieu2015deep} $\ast$ & 4 frames&1 frame \\
		Video Sequences \cite{van2017transformation} $\ast$ & 4 frames&4 frames \\

		\bottomrule
	\end{tabular}
\end{table}

\begin{figure}[t]
	\includegraphics[width=3in]{mid2}
	\caption{Intermediate image sequences visualization. \textrm{Different transformation models result in different intermediate image sequences $I_T$. Each intermediate image represent one mode of simple transformations. A sequence of these intermediate image can form more complex motion. }}
	\label{fig:intermediate}
\end{figure}
\subsection{Moving MNIST Dataset}
\label{sub:MNIST}
\paragraph{\textbf{Dataset:}}
 We first experiment on a synthetic grey video dataset: moving MNIST dataset \cite{srivastava2015unsupervised}. It consists of videos where two MNIST digits move in random directions with constant speed inside a $64 \times 64$ frame. The 64,000 training video clips and 320 testing clips are generated on-the-fly. Each video clip consists of 5 frames. Taking the first frame as input, the goal is to synthesize multiple imaginary 5-frames videos.

\paragraph{\textbf{Setup:}}
There is barely no pre-processing in our work except for normalizing all videos to be in range $[0, 1]$. We experiment on two transformation models. For convolutional transformation we set kernel size as $9 \times 9$ , and the transformation sequence length $P$ is set as 5 for both models. We generate 4 transformation sequences $\{\Phi_1,\Phi_2,\Phi_3,\Phi_4\}$ corresponding to 4 consecutive frames $\{f_1,f_2,f_3, f_4\}$ at once.

\paragraph{\textbf{Result:}}
Figure \ref{fig:result4in1} (a) illustrates the qualitative performance in moving MNIST dataset. As we can see,  frames are sharp and clear while the shape information of digits is well preserved as we expect. The difference images show that the generated transformations successfully model one motion mode so that the synthesized imaginary video has plausible consecutive motion.
Figure \ref{fig:intermediate} shows reconstructed frames and the corresponding intermediate image sequences in different transformation models.

\subsection{Synthetic 2D Shapes Dataset}
\paragraph{\textbf{Dataset:}}
We experiment our framework using a synthetic RGB video dataset: Synthetic 2D Shapes Dataset \cite{xue2016visual}. There are only three types of objects in this dataset moving horizontally, vertically or diagonally with random velocity in $[0, 5]$. All three objects are simple 2D shapes: circles, squares, and triangles. The original dataset only contains image pairs that have 2 consecutive frames. We extrepolation it to convert image pairs into video clips that have 5 frames. There are 20,000 clips for training and 500 for testing just like settings in \cite{xue2016visual}. We aim at synthesizing multiple imaginary videos each containing five consecutive frames.

\paragraph{\textbf{Setup:}}
The input image size is set as $64 \times 64$ so that we can inherit the network architecture and settings in section \ref{sub:MNIST}. The transformations applied to each color channel are set to be identical for the consistent of RGB channels.

\paragraph{\textbf{Result:}}
Figure \ref{fig:result4in1} (b) illustrates the qualitative performance in 2D shape dataset. Appearance information including color and shape is reconstructed at a satisfying level, and the motion is plausible and non-trivial.
Multiple sampling results are shown in Figure \ref{fig:multi_diff}. It is clear that sampling different $z$s lead to different imaginary videos with the same input image . Motions in those videos are notably dissimilar. Figure \ref{fig:baseline} gives an perception comparison among our framework and two baselines. The three methods are trained in same iteration. Obviously, generation from scratch as Baseline 2 needs much longer training time and $l_2$ loss criterion as Baseline 1 not only make the result lacking of diversity, but also leads to blur because of intrinsic ambiguity of image.

\subsection{UCF 101 Dataset}

\paragraph{\textbf{Dataset and setup:}}
The former datasets are both synthetic datasets. For natural scene, we experiment on UCF101 dataset \cite{soomro2012ucf101}. The dataset contains 13,320 videos with an average length of 6.2 seconds belonging to 101 different action categories. The original dataset are labeled for action recognition, but we do not employ those labels and instead we use the dataset in an unsupervised way. Videos with an average length of 6.2 seconds are cut into clips that each consists of five frames. We prepare 15,680 video clips for each category as training samples and 1,000 unseen image as testing samples. The video frames are reshaped to $128 \times 128$ and $64 \times 64$ for different resolution experiments. The convolutional kernel size is set to 16 and 9 accordingly.

\paragraph{\textbf{Result:}}
Figure \ref{fig:result4in1} (c)(d) illustrate the qualitative performance in surfing class of different resolutions. Obviously our framework produce fairly sharp frames. It successfully escapes from appearance deformation of surfer and wave. The difference images suggest that our framework can model plausible waving and surfing motions. The dynamic results seem rather realistic, so we strongly recommend a quick look at the small gif demo in supplementary material. Figure \ref{fig:wgan} shows the convergence curve of EM distance. We can see the curve decrease with training and converge to a small constant. The absolute of the constant is meaningless because the scale of EM distance varies with architecture of critic network.

\begin{figure}[t]
	\includegraphics[width=3in]{multi_diff}
	\caption{Diverse video imagination: multiple imaginary videos from same input image. \textrm{(a) denotes the input image and one imaginary video sample as reference. The first column of (b) indicates different input $z$s, the rest columns shows the difference frames of imaginary video samples minus the reference. Each row of (b) illustrates a unique imaginary video and its unique $z$. }}
	\label{fig:multi_diff}
\end{figure}
\begin{figure}[t]
	\includegraphics[width=3in]{baseline}
	\caption{Synthesis result of custom Baselines \textrm{With same fixed training iterations, our framework produce obviously better result. $l_2$ loss in baseline 1 brings blur. Baseline 2 that reconstruct pixel from noise needs much longer training time and cannot produce recognizable frames.}}
	\label{fig:baseline}
\end{figure}
\begin{figure}[t]
	\includegraphics[width=3in]{wgan}
	\caption{Curve of EM distance estimate at different steps of training. \textrm{The estimation of EM distance is done by a video critic network $C$ that is well trained. We can see that the EM distance decrease and converge with training.} }
	\label{fig:wgan}
\end{figure}

\begin{figure*}
	\includegraphics[width=7in]{comp}
	\caption{Perceptual Comparison among related works using UCF101 dataset. \textrm{The input frames are from Skijet class. The output frames are reshaped to same size for a fair visual inspection. Notice that our framework only takes one frame as input (the green square) while the rest methods take four frames as input (the grey rectangle). Our result are sharp and relatively clear while the motions of rider and skijet are recognizable and plausible.}}
	\label{fig:methods}
\end{figure*}
\subsection{Evaluation and Comparison:}
\label{sub:eva}

As shown in Table \ref{tab:task}, there is no existing work shares the completely same task settings as ours. To make fair comparison to other works and baseline, we perform both qualitative inspection and novel quantitative evaluation.

\begin{table}[t]
	\caption{quantitative evaluation comparison among related visual prediction work. \textrm{The lower $RIQA$ indicates better frame reconstruction quality. The BRISQUE score obviously varies with scenes and resolutions. $RIQA$ points out the decreasing proportion between input and output, hence successfully reflects the reconstruction quality.}}
	\label{tab:brisque}
	\begin{tabular}{cccc}
		\toprule
		Methods&\tabincell{c}{Input \\ $BRISQUE$ }&\tabincell{c}{Output \\$BRISQUE$}&\tabincell{c}{$RIQA$}\\
		\midrule
		Ours $64 \times 64$& 45.2164 &47.0168&3.98\% \\
		Ours $128 \times 128$& 35.9809 & 36.7120&\textbf{2.03\%} \\
		Baseline 1 & 45.2164 & 50.7681 &12.28\% \\
		Baseline 2 & 45.2164 & 89.2315 &97.34\% \\
		Optical Flow \cite{brox2004high} & 39.3708 & 40.8481 &3.75\% \\

		Beyond MSE \cite{mathieu2015deep} & 46.3219 & 50.0637& 9.24\% \\
		Video Sequences \cite{van2017transformation} & 39.3708&42.8834&8.92\% \\
		\bottomrule
	\end{tabular}
\end{table}

\textbf{Frame quality assessment.}
Quantitative evaluation of generative models is a difficult, unsolved problem \cite{theis2015note}. The video imagination  task is a multi-modality problem. But traditional full reference image quality assessment methods ($FIQA$) requires a precise ground truth image as reference hence they are no longer appropriate. We employ popular Blind Image Quality Assessment($BIQA$) method $BRISQUE$\cite{mittal2012no} as our non-reference quantitative evaluation metric.

Since $BRISQUE$ is based on natural
scene statistic, it is not applicable in synthetic image. we implement it on those methods that can synthesize natural scene images in UCF101 dataset \cite{brox2004high,van2017transformation,mathieu2015deep}. A key problem of employing this metric is that the scenes and resolutions of the synthesized videos may be varied, so it is unfair to make comparison among those samples directly. Fortunately, the quality of the input image can be a solid quality reference. We calculate the decreasing proportion of quality score between inputs and outputs, and take it as our assessment metric: Relative image quality assessment ($RIQA$).
\begin{equation}
RIQA = \frac{BRISQUE(Input)-BRISQUE(Output)}{BRISQUE(Input)}
\end{equation}
It is fair and reasonable because $RIQA$ eliminates the natural quality differences between scenes and resolutions while have the ability of reflecting the crucial reconstruction quality well.

As shown in Table \ref{tab:brisque}, diversity of the scenes and resolutions makes the raw BRISQUE score not comparable, but the $RIQA$ tells the reconstruction quality change. We can see that our framework outperform other methods, and the poor performances of baselines suggest the architecture of our framework is reasonable. In addition, our framework and Video Sequence\cite{van2017transformation}, that are based on transformation space, do produce images with better qualities than \cite{mathieu2015deep}, which reconstruct frames from scratch.

Table \ref{tab:para} shows the results when we change the hyper-parameters and some model settings, including the number of parameters $K$ forming transformation, the sequence length of transformation $P$ for reconstructing one frame, and the type of transformations. The results demonstrate that our framework is overall robust to those choice. It seems that affine transformation model with transformation sequence length $P=5$ can achieve the best performance.

\begin{table}[t]
	\caption{Analysis of the settings of models and hyper-parameters. \textrm{$K$ refers to the number of parameters forming transformation. $P$ refers the sequence length of transformation for reconstructing one frame.}}
	\label{tab:para}
	\begin{tabular}{cc}
		\toprule
		Settings& $RIQA$\\
		\midrule
		\tabincell{c}{affine transformation \\ with $K = 6$ and $P=5$}& \textbf{2.03\%}\\
		\tabincell{c}{affine transformation \\ with $K = 6$ and $P=10$}&4.79\%\\
		\tabincell{c}{convolutional transformation \\ with $K = 8 \times 8$ and $P=5$ }&4.03\%\\
		\tabincell{c}{convolutional transformation \\ with $K = 16 \times 16$ and $P=5$ }&4.01\%\\
		\bottomrule
	\end{tabular}
\end{table}
\textbf{Qualitative inspection.} Figure \ref{fig:methods} shows the perceptual comparison between our framework and three competing methods \cite{brox2004high,van2017transformation,mathieu2015deep} that also experimenting on UCF101 dataset. Our framework produces four frames conditioned on one frame while other methods take a sequence of four frames as inputs. The simple optical method \cite{brox2004high} fails due to the strong assumption of constant flow speed, yet it perform relatively better in quantitative evaluation because the image get weird but still maintain sharp. Beyond MSE \cite{mathieu2015deep} maintains some appearance but still struggles in deformation and blur. The transformation-based model \cite{van2017transformation} provides fairly recognizable result but
also gets blurry. Considering \cite{van2017transformation} actually takes four frames as input and aims to predict future frames, the motion looks less consecutive and convincing. Our framework synthesizes sharp and recognizable frames, and the dynamic scene looks realistic and plausible. The motion in our result (wave raising) is not identity to motion in the real video (wave falling), this is because the intrinsic ambiguity of one simgle image.
Notice that the yellow symbols on the bottom turn to pieces in our framework while in \cite{van2017transformation} it remains still. We believe this is because \cite{van2017transformation} splits frame into patches so gains better description of patch variance.

\textbf{Failure Case.} A typical failure case in affine transformation model is that the motions between frames are plausible yet unexpected black pixels appear somewhere in the frames. We think this is caused by the empty pixels in intermediate images after applying affine transformations. In convolution model, one common failure mode is that some part of the objects lack resolution while the silhouettes remain recognizable. We believe a more powerful merge network would be a promising solution in both cases,and we leave this for future work.
\section{Conclusion}
In this paper, we have presented a new framework to synthesize multiple videos from one single image. Specifically, our framework uses transformation generation to model the motions between frames, and reconstructs frames with those transformations in a volumetric merge network. We also present a novel evaluation metric to assess the reconstruction quality. We have demonstrated that our framework can produce plausible videos with state-of-the-art image quality on different datasets.

redpajama_set_name: RedPajamaArXiv


text:
\section{To be added}

Mention somewhere that they are also constant in space mimicking the procedure for unimodular [cite work with Chris].

Must add that $\alpha$ can be negative, just like $a^2$ in this approach. So $\Lambda$ can be negative, $G_N$ can be negative, even $c^2$ can be negative.

Related to this we find that putting the constraint in the form ***, may involve square roots, and so plus and minus,
which is taken into account in the range? What about branch cuts if its fractions?

\section{Semi-classical corrections}
The semiclassical limit is reassuring, but it will comport concomitant fluctuations.

Heisenberg uncertainty principle. D