SANDWICH, MA - Spaulding Rehabilitation Hospital Cape Cod has received a three-year accreditation for its inpatient rehabilitation and stroke specialty programs from CARF International, an independent, nonprofit accrediting body whose mission is to promote the quality, value, and optimal outcomes of services provided at rehabilitation facilities.
In addition to reviewing SCC's overall inpatient rehabilitation program, surveyors evaluated SCC's stroke program. "A significant portion of our patients need our expertise as a result of stroke, so the survey was an opportunity to make a deeper assessment of this program, says Daina Juhansoo, DPT, Director of Inpatient Rehabilitation. A stroke specialty program must use evidence-based treatments to minimize the patient's impairments and restrictions on activity, maximize quality of life, and assist both patient and family to promote their health over the lifespan.
SCC's three-year accreditation is the highest CARF awards and indicates that the hospital has undergone a rigorous, on-site peer review process, demonstrating its commitment to offering programs and services that are measurable, accountable, and of the highest quality.
Spaulding Rehabilitation Hospital Cape Cod (SCC) is a 60-bed acute rehabilitation hospital providing advanced inpatient and outpatient rehabilitation to residents of the Cape, Islands and South Shore. SCC is the only provider of hospital-level rehabilitation on Cape Cod and includes outpatient centers in Sandwich, Yarmouth, Orleans, and Plymouth, and a dedicated pediatric outpatient center in Sandwich. SCC is a member of The Spaulding Rehabilitation Network (SRN) and Partners HealthCare. SRN also includes Spaulding Rehabilitation Hospital, its main campus the new 132-bed facility in Charlestown, which is a national model for environmental and inclusive design; Spaulding Hospital Cambridge; two skilled nursing facilities; and 24 outpatient sites throughout Eastern Massachusetts. Spaulding strives to continually update and improve its programs to offer patients the latest, high-quality care through its leading, expert providers. Spaulding Boston is a teaching hospital of Harvard Medical School and the official rehabilitation hospital of the New England Revolution. Spaulding Boston is the only rehabilitation hospital in New England continually ranked since 1995 by U.S. News and World Report in its Best Hospitals survey with a #5 ranking in 2016-2017. For more information, please visit www.spauldingrehab.org.


It was great to have a visit today from former Head Boy Juan José Garcia (p. 2008-2009 n. 1991). Juan José read the 9th lesson at the Carol Concert as part of the 10th anniversary celebrations and wanted to come and see the new facilities at ECI. We are looking forward to seeing Juanjo and his year at the 10 year reunion in the summer.


text:
Lisa Marie Presley Is Buried in Same Place That Haunted Her All Her Life
Cover Images/INFphoto.com
The 54-year-old late star, who passed away after suffering two heart attacks on January 12, spent years hating that her family graves at the back of Graceland.
AceShowbiz - Lisa Marie Presley was haunted by having her family buried in Graceland's "graveyard." The late singer, who died aged 54 on January 12 after suffering two heart attacks, will be laid to rest in her dad's famous Memphis estate after a memorial service was held in her memory on its front lawn on Sunday, January 22.
But she spent years hating that her family graves at the back of Graceland, including those of her dad's Elvis Presley and son Benjamin Keough, who took his life aged 27 in 2020, were a constant reminder of her mortality. Her fears were shared in an interview she gave to Playboy magazine in 2003, which has been unearthed since her death.
She told the publication, "How many people have a family grave in the backyard? How many people are reminded of their fate, of their mortality, every f**king day? All the graves are lined up and there's a spot there, waiting for me, right next to my grandmother."
Lisa Marie's actress daughter Riley Keough, 33, revealed through her eulogy, read by her stuntman and actor husband Ben Smith-Petersen at her mum's memorial service, the couple had a daughter. After the service, there was a procession to Lisa's final resting place in Graceland's Meditation Garden.
The late singer's grey marble sarcophagus sat next to her burial spot in the same spot as her son Benjamin's, and photos of them together were placed in front of their monuments. Benjamin's resting place had to be moved slightly to accommodate his mum's.
Lisa Marie's body was on display to the public during the celebration of life at the family's estate, livestreamed at 9 A.M. Several of the Presley family were killed by heart conditions, including Elvis, who died in 1977 aged 42 from cardiac arrhythmia with ventricular fibrillation, a condition where the heart beats abnormally and then stops.
Lisa Marie's grandmother, Gladys Love Presley, also died from heart failure, at the age of 46, and her grandad, Vernon Elvis Presley, died at 63 from cardiac arrest. Mum-of-four Lisa Marie left Elvis' entire estate in his will when he died in 1977 aged 42, and she took control of the trust aged 25 in 1993.
Elvis was worth only around $5 million when he died, but Lisa Marie brought in millions by turning Graceland into a tourist attraction and setting up Elvis Presley Enterprises to cash in on his image and legacy.
Alec Baldwin Won't Be Charged for Shooting 'Rust' Director
Video of Jenna Ortega and 'Touchy' Percy Hynes White Goes Viral Amid Sexual Assault Allegations
Lisa Marie Presley's Pal Says Late Star Was Estranged From Mom Priscilla When She Changed Her Will
Lisa Marie Presley Was in Debt Before Death, Planned to Cash In $25M Insurance Policy to Cover It
Lisa Marie Presley's Ex-Husband Michael Lockwood 'Left in Dark' Over Singer's Estate

redpajama_set_name: RedPajamaCommonCrawl


Healthcare is in the midst of a most fundamental transformation, with mobile technologies invading the scene. 83% of clinicians already confirm usage of mobile technologies to provide patient care. Venture capitalists are turning away from social networks and gaming to invest heavily in digital health, which offers an estimated $60 billion in market opportunities. The time has come to jump on the mHealth bandwagon. Do it wisely. Here are some of the most important facts that will help you make weighted, conscious mHealth decisions.
What are the benefits of mobile technology in healthcare?
self-testing (including sensor triggering for heart rate monitoring).
Medication compliance. Mobile technology helps patients follow medication orders. It can also predict when a prescription runs out and remind the patient to get a timely refill, as well as provide regular dosage reminders and delayed reminders.
Medication accuracy. Mobile apps are designed to help double-check important details and increase patient's safety, in an environment where clinical instructions and prescriptions add up to a lot of paperwork.
Access to health information. Mobile health technologies helps patients view, download, and share their health information electronically.
For health research side of the industry, mobile tech brings in the ability to track health behaviors over huge time and space laps. With the help of big data technologies, federal agencies and healthcare organizations are enabled to cross-reference data and build a database of trends that would improve medical practice. As we see, the benefits across various domains are multiple. However, as any innovation, mHealth has its own red flags.
What are the potential risks of mobile technology in healthcare?
The main concern for both healthcare professionals and patients is data security. Health organizations are obliged to keep protected patient information (PHI) safe in a mobile environment. One challenge for health care systems is the so-called BYOD, "bring your own device", which means employees are using personal mobile devices at work. As a matter of fact, almost 89% of U.S. healthcare professionals use their personal mobile devices for work. But when it comes to security, 41% of healthcare employees do not protect their smartphones with a password, and 53% access unsecured wireless networks. Apps, both free and paid, can send unencrypted data to advertisers and third-party data analytics firms. In fact, around 50% of mobile applications for healthcare industry fail to use encryption technology to protect data during transmission and storage of sensitive information, have vague or non-existent privacy policies, and do not comply with HIPAA, Health Insurance Portability and Accountability Act.


It has become almost a ritual to see the beautiful and very talented Mouni Roy on Bigg Boss. The actress will be appearing on the show yet again and will share the stage with Salman Khan.
According to reports in the media, the actress will be seen joining Salman on stage. She will perform on some of his hit songs and the host of the show will also join her. Apart from this Mouni will also have an entertaining segment with Salman. The channel is planning something grand for Salman on his birthday which he celebrates on December 27. Salman and Mouni will be seen grooving with each other.
For the uninitiated, the actor is fond of Mouni and she had even featured in one of the initial promos of Bigg Boss 11. Mouni also has a special place for the superstar in her heart as he is her favourite actor.


Single point adjustment, no ladder or chair needed.
Perfect for sand courts and prolonged outdoor use.
Posts can be permanently placed in the ground or you can purchase optional ground sleeves with removable caps.


text:
8 Impactful Ways to Celebrate International Women's Day 2020
What is International Women's Day, and why do we celebrate it each year?
International Women's Day is celebrated each year on March 8th. It is a global day to celebrate the social, cultural, economical and political achievements of women. This tradition is well over a century old, with the first International Women's Day gathering taking place in 1911.
International Women's Day is not specific to any certain country, organization, or group - it's a day of celebration for everyone, everywhere, as well as a call to action for gender equality.
According to the official website for International Women's Day, it is a day for "unity, celebration, reflection, advocacy and action - whatever that looks like at a local level".
Here are 8 simple, yet impactful ways to celebrate International Women's Day this year:
1. Support the theme of International Women's Day 2020 and raise awareness by posting to social media.
© Photograph: Sarah Koester / EyeEm
The theme this year is Each for Equal, meaning that each of us can play a part in creating a more equal world. You can post to social media with this year's hashtag #EachforEqual, and strike the pose that goes along with it.
2. Donate to your local women's shelter, whether in the form of a monetary donation, clothing, supplies, or your time.
3. Support local, female-run businesses. Try to see how many female-run businesses you can support on International Women's Day, as you go about your day!
4. Watch Ted talks given by women. Ted has a list of 10 talks by women that everyone should watch.
5. Read impactful books written by women. Here are a few to get you started:
The Joy Luck Club by Amy Tan
The Color Purple by Alice Walker
We Should All Be Feminists by Chimamanda Ngozi Adichie
6. Celebrate women in film. Of the top 100 highest grossing films from 2019, only 12% were directed by women. And that is still a huge jump from the measly 7% out of 250 in 2016. As far as female leads, women are still being represented in Hollywood, percentage wise (43%), less than the amount of women in the population of the world (49.6%).
Why not celebrate by enjoying some female-directed films, and films with female leads this year? Here are a couple to begin with:
Suffragette, directed by Sarah Gavron. About the UK's suffragette movement in 1912, one of the most important social movements in history.
Hidden Figures, With Taraji P. Henson, Octavia Spencer & Janelle Monae in leading roles. Th story of a trio of female NASA scientists who helped launch the first American astronaut into space.
7. Spend time with your mom, or with other women whom you admire.
8. Do something kind for a woman you love. Even the smallest gesture can make a difference in someone's day, or week.
This March 8th, let's reflect on what the brave and strong women and allies of the past have accomplished to bring about the greater equality that we enjoy today. And, let's remember to continue to empower and support women today as we strive to achieve gender parity in our world. Let's do our part to ensure that the future for girls around the world is even brighter, more safe, more rewarding, and equal.
Amy Rawlings
Rice Love
https://www.internationalwomensday.com/About
https://www.internationalwomensday.com/Theme
https://www.workflowmax.com/blog/10-excellent-ways-to-celebrate-international-womens-day
https://womenandhollywood.com/resources/statistics/
Back to The Journal


Other Showcase your most recent craft!
Woodcraft DIY, House Of Horror with a $200 budget.
Papercraft Harry Potter Ideas for Halloween Front Porch?
Other Halloween pumpkin, made from matchstick burns.
I think I don't have more imagination!


text:
\section*{Acknowledgements}

This work was granted access to the HPC resources of IDRIS under the allocation 2020-AD011012227 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the ``Investissements d'avenir'' program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). JP was supported in part by the Louis Vuitton/ENS chair in artificial intelligence and the Inria/NYU collaboration. \revision{We thank the reviewers for useful comments.}
\section{Discussion}
\label{sec:discussion}

CCVS{} is on par or better than the state-of-the-art on standard benchmarks, uses less computational resources, and scales to high resolution. Training neural networks is environmentally costly, due to the carbon footprint to power processing hardware~\citep{strubell2019energy, dhar2020the}. Methods sparing GPU-hours like ours are crucial to make AI less polluting~\citep{strubell2019energy, lacoste2019quantifying}, and move from a ``Red'' to a ``Green'' AI~\citep{schwartz2020green}.
Future work will include exploring new codebook strategies and synthesis guided by textual information.

\paragraph{Limitations.} \revision{CCVS{} uses a complex architecture and a two-stage training strategy. Simplification of both is an interesting direction for improving the method. Moreover, CCVS{} lacks global regularization of motion (flow computed on pairs of timesteps), and its efficiency relies on recycling context information such that synthesizing content from scratch (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, no input frame given) remains difficult.}

\paragraph{Broader impact.} The increased accessibility and the many controls CCVS{} offers could accelerate the emergence of questionable applications, such as ``deepfakes'' (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, a video created from someone's picture and an arbitrary audio) which could lead to harassment, defamation, or dissemination of fake news.
On top of current efforts to automate their detection~\citep{nguyen2019deep}, it remains our responsibility to grow awareness of these possible misuses.
Despite these worrying aspects, our contribution has plenty of positive applications which outweigh the potential ethical harms. Our efficient compression scheme is a step in the direction of real-time solutions: \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, enhancing human-robot interactions, or improving the safety of self-driving cars by predicting the trajectories of people and vehicles nearby.
\section{Experiments}
\label{sec:experiments}
We assess the merits of CCVS{} in the light of extensive experiments on various video synthesis tasks.

\paragraph{Datasets.} BAIR Robot Pushing~\citep{ebert2017self} consists of 43k training and 256 test videos of a robotic arm interacting with objects from a fixed viewpoint. A high-resolution version has recently been released. We manually annotate, in $500$ frames, the $(x,y)$ location of the arm in image space to train a position estimator, which we use for state-conditioned synthesis. To account for real world scenarios, we evaluate on Kinetics-600~\citep{carreira2018a}, a large and diverse action-recognition dataset with approximately 500K videos. We also test our method on AudioSet-Drums~\citep{gemmeke2017audio} for sound-conditioned synthesis on music performance, containing 6k and 1k video clips in train and test splits respectively. Other datasets and tasks are covered in Appendix~\ref{sec:additional-results}.

\paragraph{Metrics.} We use the Fréchet video distance (FVD)~\citep{unterthiner2018towards} which measures the distribution gap between real and synthetic videos in the feature space of an Inception3D network~\citep{carreira2017quo} pretrained on Kientics-400~\citep{kay2017the}. \revision{It estimates the visual quality and temporal consistency of samples as well as the diversity in unconditioned scenarios. In conditioned ones, we use another metric for diversity (DIV) which is the mean pixel-wise distance among synthetic trajectories conditioned on the same input.} For near-deterministic motions (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, in reconstructions, or constrained tasks), there is a one-to-one mapping between real video frames and synthetic ones, and we include pairwise image quality assessments: the structural similarity index measure (SSIM)~\citep{wang2004image} which evaluates a per-frame conformity (combination of luminance, contrast and structure), and the peak signal-to-noise ratio (PSNR) which is directly related to the root mean squared error. For each metric, we compute the mean and standard deviation (std) over $5$ evaluation runs (2 for Kinetics due to its voluminous 50k video test set). For clarity, the std value is shown only when it is greater than the reported precision.

\paragraph{Training details.}
All our models are trained on 4 Nvidia V100 GPUs (32GB VRAM each), using ADAM~\citep{kingma2014adam} optimizer, for multiple 20 hour runs. We adapt the batch size to fill the memory available. We use a learning rate of $0.02$ to train the autoencoder, and exponential moving average~\citep{yaz2019unusual} to obtain its final parameters. We use weighting factors $(1, 10, 1, 1)$ and $0.25$ for $(\lambda_q, \lambda_r, \lambda_a, \lambda_c)$ and $\beta$ in Equations (\ref{eq:ae}) and (\ref{eq:q}) respectively. We use a learning rate of $10^{-5}$ to train the transformer.

\subsection{Ablation study}
\label{sec:ablation}
\input{neurips2021/tables/bair-ae-ablation-256}
\input{neurips2021/tables/bair-transformer-ablation-256}

We conduct an ablation study of CCVS{} to show the individual contribution of the key components of the proposed autoencoder (Table~\ref{tab:ae-ablation}), transformer (Table~\ref{tab:transformer-ablation}), and the effect of compression (Figure~\ref{fig:compression}).

First, we fix the transformer and observe the incremental improvements in synthesis quality when adding self- and context-recovery modules to the autoencoder (Table~\ref{tab:ae-ablation}). In particular, $L_1$ loss in the feature space of a $\VGG$ net~\citep{simonyan2015very} produces sharper videos than using the same loss in the RGB space alone. Predicted frames using image and temporal discriminators ($\Delta_i$ and $\Delta_t$) display greater realism and temporal consistency. The flow module $\mathcal{F}$ significantly improves the performance on all metrics by allowing context frames to guide the reconstruction of synthetic ones. The self-supervision of $\mathcal{F}$, the use of larger context windows, and longer training times, further improve the quality of the synthesis. $\Delta_t$ seems to deteriorate the FVD at first, but when all modules are combined it improves FVD by almost a factor $2$ as it encourages better temporal consistency in the presence of context.

We fix the autoencoder, and compare different architectures and sampling strategies for the transformer (Table~\ref{tab:transformer-ablation}). Increasing the model capacity by adding layers and increasing expressivity (number of attention heads) along with a spatio-temporal decomposition of positional embeddings (shown in Figure~\ref{fig:gpt}, detailed in supplementary material) yields small improvements. Top-$k$ sampling is beneficial for stochastic tasks (video prediction and unconditional synthesis), whereas always selecting the most probable tokens results in little to no motion. Guiding temporal dynamics with state-conditioned synthesis reduces the advantage of sampling by narrowing down possible outcomes.

Finally, we explore the effect of compression (in terms of the number of tokens for each frame) on synthesis speed (FPS) and quality (FVD) of reconstructed and predicted videos (Figure~\ref{fig:compression}). We use a compression of $64$ tokens in our default setup since it gives the best FVD while retaining a reasonable speed. The critical drop of FPS for low compression ratios is due to the pairwise consideration of all input tokens in $\mathcal{T}$. Note that using a smaller temporal window in $\mathcal{T}$ may allow additional speed-ups.

\subsection{Quantitative and qualitative studies}
\label{sec:qua}
\input{neurips2021/tables/bair-prediction-64}
\input{neurips2021/tables/bair-qualitative-256}

\textbf{BAIR.} For future video prediction on BAIR (Table~\ref{tab:bair-prediction-64}), CCVS{} trained at $64 \hspace{-0.15em}\times\hspace{-0.15em} 64$ resolution (\textit{low res.}) is on par with the best method (L-size version of~\citep{weissenborn2020scaling}), but requires much less computing resources, and outperforms~\citep{weissenborn2020scaling} under similar resources. We also propose \textit{high res.} CCVS{} which is not strictly comparable to the prior arts as we use $256 \hspace{-0.15em}\times\hspace{-0.15em} 256$ image resolution for training and test, and resize the synthesized frames to $64 \hspace{-0.15em}\times\hspace{-0.15em} 64$ for computing FVD. However, using this variant demonstrates the performance gains that can arise by scaling CCVS{}. We additionally address point-to-point synthesis (with the end frame as a video-level annotation) and state-conditioned synthesis (with the estimated 2D position of the arm as a frame-level annotation). Point-to-point synthesis is more difficult than video prediction: Not only does it requires realistic video continuations, but also ones which explain the end position of all visible objects. Hence, FVD score is constant despite the additional input. Still, this yields better SSIM for mid-point and one-before-last frames. State-conditioned synthesis improves on FVD and mid-synthesis SSIM as motion becomes near-deterministic. Some synthetic frames for these tasks are shown in Figure~\ref{fig:bair-qualitative-256}. CCVS{} creates plausible high-quality videos in various settings, and true interactions with objects compared to previous attempts~\citep{menapace2021playable} at the same resolution. \revision{}

\input{neurips2021/tables/kinetics-prediction-64}
\input{neurips2021/tables/kinetics-qualitative-64}
\input{neurips2021/tables/drum-audio-64}
\input{neurips2021/tables/div}

\textbf{Kinetics.} CCVS{} ranks second on Kinetics video prediction benchmark. Kinetics contains more diversity than BAIR, and the reconstruction is thus more difficult. A solution~\citep{ramesh2021zero} is to increase the codebook size (Table~\ref{tab:kinetics-codebook-64}) but it stops translating into better prediction FVD at some point. We also try removing the quantization step (equivalent to an infinite codebook), and directly regressing latent features with $\mathcal{T}$ (instead of ranking the likelihood of possible tokens). It allows better reconstructions, yet the prediction FVD is high. \revision{Figure~\ref{fig:kinetics-qualitative-64} shows examples of synthetic continuations conditioned on 5-frame unseen test sequences. CCVS{} produces realistic and temporally coherent outputs which display various types of motion (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, body, hand, camera).}

\textbf{AudioSet-Drums.} CCVS{} achieves top performance on sound-conditioned video synthesis on AudioSet Drums (Table~\ref{tab:drum-sound-64}). \revision{Figure~\ref{fig:div} shows quantitative and qualitative insights on the diversity of the synthetic trajectories conditioned on the same input for the three datasets. The diversity metrics (DIV) computed on whole frames is lower on AudioSet Drums than on the other two datasets. This is explained by the fact that motion is quite repetitive and involves a limited portion of the frame. The same metric on moving parts only, and the end position of the drummer's hand and upper left cymbal in qualitative samples, show the diversity of synthetic trajectories. An ablation of CCVS{} with/without audio guidance as well as more qualitative results on diversity can be found in Appendix~\ref{sec:additional-results}.}

\section{Introduction}
\label{sec:introduction}

Feeding machines with extensive video content, and teaching them to create new samples on their own, may deepen their understanding of both the physical and social worlds. Video synthesis has numerous applications from content creation (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, deblurring, slow motion) to human-robot interaction (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, motion prediction). Despite the photo-realistic results of modern image synthesis models~\citep{karras2020analyzing}, video synthesis is still lagging behind due to the increased complexity of the additional temporal dimension.

An emerging trend is to use autoregressive models, for example transformer architectures~\citep{vaswani2017attention}, for their simplicity, and their ability to model long-range dependencies and learn from large volumes of data~\citep{devlin2019bert, brown2020language}. \revision{First introduced for natural language processing (NLP) and then succesfully applied to visual data~\citep{dosovitskiy2021an},} the strength of transformers is grounded in a self-attention mechanism which considers all pairwise interactions within the data. The price to pay is a computational complexity which grows quadratically with the data size, which itself depends linearly on the temporal dimension and quadratically on the spatial resolution in the image domain. Although there have been some efforts to reduce the complexity of self-attention~\citep{kitaev2020reformer, katharopoulos2020transformers, choromanski2021rethinking}, using such methods directly on visual data is still limited to low resolutions and impractical without considerable computational power~\citep{chen2020generative, weissenborn2020scaling}.

Some recent works~\citep{esser2020taming, rakhimov2020latent} address this problem by using an autoencoder to compress the visual data, and apply the autoregressive model in the latent space. This greatly reduces the memory footprint and computational cost, yet, the greater the compression, the harder it is to faithfully reconstruct frames. The corresponding trade-offs may undermine the practical usability of these approaches. GANs~\citep{goodfellow2014generative} mitigate this issue by ``hallucinating'' plausible local details in image synthesis~\citep{esser2020taming}. But latent video transformers~\citep{rakhimov2020latent} decode frames independently, which prevents local details from being temporally coherent. Hence, using GANs in this setting may result in flickering effects in the synthesized videos.

We follow the problem decomposition from~\citep{rakhimov2020latent, esser2020taming}, but introduce a more elaborate compression strategy with CCVS{} (for \emph{Context-aware Controllable Video Synthesis}), an approach that takes advantage of ``context'' frames (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, both input images and previously synthesized ones) to faithfully reconstruct new ones despite lossy compression. As shown in Figure~\ref{fig:skip-ae}, CCVS{} relies on optical flow estimation between context and new frames, within temporal skip connections, to let information be shared across timesteps. New content, which cannot be retrieved from context, is synthesized directly from latent compressed features, and adversarial training~\citep{goodfellow2014generative} is used to make up realistic details. Indeed, information propagates in CCVS{} to new frames as previously synthesized ones become part of the context. Like other video synthesis architectures based on autoregressive models~\citep{weissenborn2020scaling, rakhimov2020latent}, CCVS{} can be used in many tasks besides future video prediction. Any data which can be expressed in the form of a fixed-size sequence of elements from a finite set (\textit{aka}, tokens) can be processed by a transformer. This applies to video frames (here, via compression and quantization) and to other types of data. Hence, one can easily fuse modalities without having to build complex or task-specific architectures. This idea has been used to control image synthesis~\citep{esser2020taming, ramesh2021zero}, and we extend it to a variety of video synthesis tasks by guiding the prediction with different annotations as illustrated in Figure~\ref{fig:gpt}. Code, pretrained models, and video samples synthesized by our approach are available at the url \url{https://16lemoing.github.io/ccvs}. Our main contributions are as follows:
\begin{enumerate}[noitemsep,topsep=0pt,left=7pt]
    \item an optical flow mechanism within an autoencoder to better reconstruct frames from context,
    \item the use of ancillary information to control latent temporal dynamics when synthesizing videos,
    \item a performance on par with or better than the state of the art, while being more memory-efficient.
\end{enumerate}
\section{Context-aware controllable video synthesis}

\subsection{Overview of the proposed approach}

\begin{figure}
    \centering
    \includegraphics[width=0.60\linewidth, valign=c]{neurips2021/figures/auto-encoder-31.pdf}
    \includegraphics[width=0.20\linewidth, valign=c]{neurips2021/figures/auto-encoder-32.pdf}
	\caption{Proposed architecture. Here $X_c$ and $X_s$ respectively stand for the (input) context and (output) synthesized video. Learnable encoder and decoder modules $\mathcal{E}$ and $\mathcal{D}$ are linked by a learnable flow estimation module $\mathcal{F}$ ensuring spatio-temporal consistency between context and synthesized frames. The architecture is doubly autoregressive, with a transformer $\mathcal{T}$ responsible for predicting the features $Z_s$ associated with future frames $X_s$ from the features $Z_c$ associated with context frames $X_c$, and a simple, parameterless, ``shift-and-add'' module $\mathcal{S}$ updating $X_c$ as each new frame is generated. The architecture is trained in two steps: The parameters of $\mathcal{E}$, $\mathcal{D}$ and $\mathcal{F}$ are first estimated (without any future forecasting from the transformer) in an adversarial manner using two discriminators $\Delta_i$ and $\Delta_t$ to ensure that the frames synthesized are both realistic ($\Delta_i$) and temporally coherent ($\Delta_t$). The two discriminators are then discarded, and the parameters of the transformer $\mathcal{T}$ are estimated with $\mathcal{E}$, $\mathcal{F}$ and $\mathcal{D}$ frozen. At inference time, the transformer is used only once, latter frames being estimated in an autoregressive manner by the quadruple $\mathcal{D}$ $\mathcal{S}$, $\mathcal{E}$, $\mathcal{F}$. See text for more details.}
	\label{fig:skip-ae}
\end{figure}

\label{sec:overview}
\revision{We consider video data, such that an individual frame $x$ is natively an element of $\mathcal{X}=\mathbb{R}^{H \times W \times 3}$, which can be encoded as some feature $z$ in $\mathcal{Z}=\mathbb{R}^{h \times w \times F}$ with reduced spatial resolution and increased number of channels. We assume that we are given $X_c$ in $\mathcal{X}^m$ corresponding to $m$ successive frames, and our goal is to {\em synthesize} a plausible representation of the following $n$ frames $X_s$ which lies in $\mathcal{X}^n$. Given $X_c$, we can compute the corresponding features $Z_c$ in $\mathcal{Z}^m$ using some encoder $\mathcal{E}:\mathcal{X}\rightarrow\mathcal{Z}$ on each frame individually, then use an autoregressive model (a transformer coupled with a quantization step in our case) to predict features $Z_s$ in $\mathcal{Z}^n$ formed by the $n$ features corresponding to time ticks $m+1$ to $m+n$. (Note that the values of $m$ and $n$ can be arbitrary, using [by now] traditional masking and sliding window techniques.) These features can finally be converted one by one into the corresponding frames $X_s$ using some decoder $\mathcal{D}:\mathcal{Z}\rightarrow \mathcal{X}$.}

\revision{The overall approach is illustrated by Figure~\ref{fig:skip-ae}. The use of a quantizer over a learned codebook in our implementation complicates the architecture a bit, but has several advantages, including the reuse of familiar NLP technology~\citep{vaswani2017attention} and, perhaps more importantly, affording simple mechanisms for handling different types of inputs (from images to sound for example~\citep{chen2020generative, huang2018music}) and the intrinsically uncertain and multimodal nature of future prediction (by sampling different codebook elements according to their likelihood~\citep{holtzman2020curious}).
Concretely, this choice simply amounts to inserting in our architecture, right after the encoder $\mathcal{E}$, a nearest-neighbor quantizer $\mathcal{Q}:\mathbb{R}^F\rightarrow\llbracket 1,q\rrbracket$ parameterized by an $F \times q$ codebook which, given the embedding $z$ of a frame, returns a $h \times w$ matrix of corresponding {\em tokens}, that is, the indices of the closest entry in the codebook for feature vector $z_{i,j}$ for all spatial locations $(i, j)$ in $\llbracket 1,h\rrbracket\times\llbracket 1,w\rrbracket$. We abuse notation, and identify $\mathcal{Q}$ with its parameterization by this codebook, so we can optimize over $\mathcal{Q}$ just as we optimize over $\mathcal{E}$ instead of naming explicitly its parameters in the rest of this presentation. An ``un-quantizer'' $\mathcal{U}: \llbracket 1,q\rrbracket\rightarrow \mathbb{R}^F$, also parameterized (implicitly) by the codebook and associating with each token the corresponding entry of the codebook, is also inserted right before the decoder $\mathcal{D}$.
In this setting, $\mathcal{T}:\llbracket 1,q\rrbracket^{m\times h\times w} \rightarrow \llbracket 1,q\rrbracket^{n\times h \times w}$ takes as input a sequence of tokens, and outputs the tokens for subsequent frames, each one of them chosen among $q$ possibilities as either the one with the highest score, or drawn randomly from the top-$k$ scores to account for the multimodal nature of future forecasting (here $k\le q$ is some predefined constant, see~\citep{holtzman2020curious} for related approaches).}

\revision{The elements of the architecture described so far are by now (individually) rather classical, with learnable, parametric functions $\mathcal{E}$, $\mathcal{D}$, $\mathcal{Q}$ and $\mathcal{T}$. Besides putting them all together, and as detailed in the rest of this section, we add several original elements: {\bf (a)} The encoding/decoding scheme is improved by the use of two discriminators, $\Delta_i$ and $\Delta_t$ respectively, trained in an adversarial manner to ensure that the predicted frames are realistic and temporally consistent. {\bf (b)} The context frames $X_c$ are themselves updated each time a new frame is predicted in an autoregressive manner (iteratively fill the sequence up to some predefined capacity, then shift to the left, forgetting the first frame and adding the latest synthetic one on the right). {\bf (c)} The encoder and decoder are linked through a learnable {\em flow module} $\mathcal{F}$, allowing the context frames to guide the prediction of the synthetic ones. {\bf (d)} Additional {\em control variables}, ranging from object trajectories to audio tracks, can be used in the form of sequence- or frame-level annotations to drive the synthesis by adding the corresponding tokens to the ones passed on to the autoregressive model~$\mathcal{T}$.}

\revision{We detail in this section the concrete components of the approach sketched above, including the autoencoder and quantizer architectures and their training procedure (Section~\ref{sec:ae}), and the implementation of the autoregressive model by a transformer~\citep{vaswani2017attention}, illustrated in Figure~\ref{fig:gpt}, which we adapt to account for outside control signals (Section~\ref{sec:transformer}), once again with the corresponding procedure. Further architectural choices are also detailed in Appendix~\ref{sec:architecture}.}

\subsection{First stage: training the context-aware autoencoder and the quantizer}
\label{sec:ae}

\paragraph{Architecture.} \revision{$\mathcal{E}$ and $\mathcal{D}$ respectively decreases and increases the spatial resolution by using convolutional residual blocks~\citep{he2016deep}, with $(r_k)_{k\in \llbracket 1;K \rrbracket}$ the $K$ corresponding resolution levels ($r_k=h_k \times w_k$). It is common practice to augment, as in U-Net~\citep{ronneberger2015unet}, the autoencoder with long skip connections between $\mathcal{E}$ and $\mathcal{D}$ to share information across the two models at these intermediate levels and escape the lossy compression bottleneck. Although we cannot apply this directly to our setting since information only flows through $\mathcal{D}$ for predicted timesteps, such skip connections can be established from the encoding stage of a context frame $x_c$ to the decoding stage of a new frame $x_s$ (resulting from features $z_s$). Similar mechanisms~\citep{ebert2017self, denton2018stochastic} have been proposed in the past for video synthesis but they are only copying static background features from a single context frame.
We follow works on semantic segmentation~\citep{gadde2017semantic} and face frontalization~\citep{wei2020learning} and use a flow module $\mathcal{F}$ to warp features and produce temporally consistent outputs despite motion. We extend this to multi-frame contexts, with significant performance gains and no additional parameter to be learned.}

\revision{Concretely, let $e_c^k$ be features being encoded from $x_c$, and $d_s^k$ features being decoded from $z_s$ at a given intermediate resolution~$r_k$. We first compute all intermediate context features $e_c^k$ by applying $\mathcal{E}$ to $x_c$. We then progressively decode features $d_s^k$ for the new frame from low ($k=1$) to high resolution ($k=K$) by iterating over the following steps: {\bf (a)} apply one decoding sub-module to get $d_s^k$ from $d_s^{k-1}$, {\bf (b)} use $\mathcal{F}$ to refine the optical flow $f_c^k$ (in $\mathbb{R}^{2 \times r_k}$) which estimates the displacement field from $e_c^k$ to $d_s^k$ (as a proxy to the one from $x_c$ to $x_s$), and a fusion mask $m_c^k$ (in $\mathbb{R}^{1 \times r_k}$) which indicates the expected similarity between aligned features $e_c'^k=\warp(e_c^k, f_c^k)$ and $d_s^k$ (also as a proxy for the one in image domain) with $\warp$ corresponding to a standard warping operation, {\bf (c)} use $e_c'^k$ and $m_c^k$ to update $d_s^k$ with context information (see update rule~(\ref{eq:update}) below), {\bf (d)} move to resolution level~$r_{k+1}$ by going back to {\bf (a)}. We note that $\mathcal{F}$ estimates $f_c^k$ and $m_c^k$ in a coarse-to-fine fashion by refining $f_c^{k-1}$ and $m_c^{k-1}$ (see Appendix~\ref{sec:architecture} for further details).} Temporal skip connections at a given resolution level $r_k$ are defined as the following in-place modification of $d_s^k$:
\begin{equation}
    d_s^k=\sigma(m_c^k) \otimes d_s^k+(\mathbf{1}-\sigma(m_c^k)) \otimes e_c'^k,
\label{eq:update}
\end{equation}
where $\sigma$ is the Sigmoid function, and $\otimes$ the element-wise product. \revision{We note that update rule~(\ref{eq:update}) is quite standard for warping and fusing two streams of spatial information~\citep{wang2018video, hao2018controllable, gadde2017semantic}.} For concrete implementation of $\mathcal{F}$, we build upon LiteFlowNet~\citep{hui2018liteflownet}, an optical flow estimation model which also combines pyramidal extraction and progressive warping of features. For simple integration into our framework, we use features from $\mathcal{E}$ and $\mathcal{D}$ both in the mask and flow estimation, and in the update~(\ref{eq:update}). This process readily generalizes to multi-frame contextual information (see Appendix~\ref{sec:multi} for details). \revision{Similar to spatial transformers~\citep{jaderberg2015spatial}, the warping operation $\warp$ is differentiable. As a result, gradients from the training losses can backpropagate from $\mathcal{D}$ to $\mathcal{E}$ through $\mathcal{F}$. This allows end-to-end training of the autoencoder even with information from different timesteps in $\mathcal{E}$ and $\mathcal{D}$.}

\paragraph{Training procedure.} The global objective is the linear combination of four auxiliary ones:
\begin{equation}
\label{eq:ae}
    \mathcal{L} = \lambda_q\mathcal{L}_q + \lambda_r\mathcal{L}_r + \lambda_a\mathcal{L}_a + \lambda_c\mathcal{L}_c,
\end{equation}
namely a quantization loss ($\mathcal{L}_q$), a reconstruction loss ($\mathcal{L}_r$), an adversarial loss ($\mathcal{L}_a$), and a contextual loss ($\mathcal{L}_c$), detailed in the next paragraphs.

The codebook is trained by minimizing the reconstruction error between encoded features $z=\mathcal{E}(x)$ and quantized features $z_q=\mathcal{U}(\mathcal{Q}(z))$ (with notations introduced in Section~\ref{sec:overview}):
\begin{equation}
\label{eq:q}
    \mathcal{L}_q(\mathcal{E}, \mathcal{Q})=\|\sg(z)-z_q\|_2^2 + \beta \|\sg(z_q)-z\|_2^2,
\end{equation}
where $\sg(.)$ is the stop gradient operation which constrains its operand to remain constant during backpropagation. The first part moves the codebook entries closer to the encoded features. The second part, known as the \textit{commitment loss}~\citep{vandenoord2017neural}, reverses the roles played by the two variables.

In regions with complex textures and high frequency details, local patterns shifted by a few pixels in $x$ and its reconstruction $\widehat{x}=\mathcal{D}(z_q)$ may result in large pixel-to-pixel errors, while being visually satisfactory. We thus define the recovery objective as the $L_1$ loss in both RGB space and between features from a $\VGG$ network~\citep{simonyan2015very} pretrained on ImageNet~\citep{deng2009imagenet}:
\begin{equation}
\label{eq:recovery}
    \mathcal{L}_r(\mathcal{E}, \mathcal{Q}, \mathcal{F}, \mathcal{D})=\|x-\widehat{x}\|_1+\|\VGG(x)-\VGG(\widehat{x})\|_1.
\end{equation}

To tackle cases where information cannot be recovered from context due to occlusion, and the compressed features are insufficient to create plausible reconstructions due to lossy compression, we supplement our architecture with an image discriminator $\Delta_i$, made of downsampling residual blocks~\citep{he2016deep}, to encourage realistic outputs. $\Delta_i$ tries to distinguish real images from reconstructed ones ($\mathcal{L}_d$), while $\mathcal{E}$ and $\mathcal{D}$ fools $\Delta_i$ into assuming reconstructed images are as good as real ones ($\mathcal{L}_a$):
\vspace{-0.35cm}
\begin{align}
\begin{minipage}{.5\linewidth}
\begin{equation}
    \mathcal{L}_d(\Delta_i)=\ln(1+e^{\Delta_i(x)})+\ln(1+e^{-\Delta_i(\widehat{x})}),
\label{dis-loss}
\end{equation}
\end{minipage}%
\begin{minipage}{.5\linewidth}
\vspace{0.20cm}
\begin{equation}
    \mathcal{L}_a(\mathcal{E}, \mathcal{Q},  \mathcal{F}, \mathcal{D})=\ln(1+e^{\Delta_i(\widehat{x})}).
\label{adv-loss}
\end{equation}
\end{minipage}
\end{align}
We employ a similar strategy on sequences of consecutive frames to improve the temporal consistency using a 3D temporal discriminator $\Delta_t$, a direct extension of 2D image discriminator $\Delta_i$.

The success of our method relies on accurate motion estimation in $\mathcal{F}$, a difficult task which benefits from self-supervision~\citep{jonschkowski2020what}. Therefore, we train the autoencoder with augmented views of the input frames as context. Custom augmentations functions $\aug:\mathcal{X}\rightarrow\mathcal{X}$ include: rotation, scaling, translation, elastic deformation, and combinations of these. Augmented views are obtained by warping $x$ by the suitable flow $a_c$: $\aug(x)=\warp(x,a_c)$,
and the inverted flow $f_c$ from $\aug(x)$ to $x$ can be approximated.\footnote{Our implementation of flow inversion approximation is detailed in Appendix~\ref{sec:inversion}.} We resort to flow inversion because directly reconstructing distorted views may encourage similar defects during inference. Moreover, we balance between self-recovery and context-recovery objectives by additionally applying a blurring function $\blur:\mathcal{X}\rightarrow\mathcal{X}$ and an occlusion mask $o_c$ to the augmented frames $x_c=o_c \otimes \blur(\aug(x))$. This augmentation strategy is illustrated in Appendix~\ref{sec:augmentation}. We define the contextual loss as:
\begin{equation}
    \mathcal{L}_c(\mathcal{E}, \mathcal{Q}, \mathcal{F}, \mathcal{D})=\lVert f_c - \widehat{f}_c \rVert_2^2+\lVert o_c' - \sigma(\widehat{m}_c) \rVert_2^2,
\label{flow-loss}
\end{equation}
where $o_c'=\warp(o_c,f_c)$, and $\widehat{f}_c$ and $\widehat{m}_c$ are the flow and mask estimated by $\mathcal{F}$. In practice, this loss is applied at intermediate resolution levels $r_k$ for improved training.

\subsection{Second stage: predicting temporal dynamics with transformers}
\label{sec:transformer}

\begin{figure}
    \centering
    \includegraphics[width=0.76\linewidth, valign=c]{neurips2021/figures/transformer-21.pdf}
    \includegraphics[width=0.23\linewidth, valign=c]{neurips2021/figures/transformer-22.pdf}
	\caption{Illustration of the transformer to predict latent temporal dynamics with control-related data. For clarity, we omit the flow module $\mathcal{F}$ between $\mathcal{E}$ and $\mathcal{D}$. Input video-level annotation, frame-level annotations, and conditioning frames are encoded and quantized to form the initial token sequence, and mapped to corresponding embeddings. Subsequent frame tokens are obtained autoregressively with $\mathcal{T}$, while the ones corresponding to video- and frame-level annotations guide the prediction.}
    \label{fig:gpt}
\end{figure}

\paragraph{Architecture.} We follow~\citep{esser2020taming} and adopt an architecture similar to Image-GPT~\citep{chen2020generative} for $\mathcal{T}$. Instead of modeling a single annotated frame as in~\citep{ramesh2021zero}, we design our model to account for sequences of $N$ such frames to allow prediction of temporal dynamics controlled by ancillary information in the form of video- and frame-level annotations. We have shown, in Section~\ref{sec:ae}, how to represent a frame as a sequence of $h \times w$ tokens (indices in $\llbracket 1,q\rrbracket$) through encoding and quantization. Similar strategies can be applied to cater to other types of data, with or without compression depending on their complexity, and, thereby, turn ancillary information into tokens as well. The capacity of the transformer (the maximum sequence length it can process) is $L=l_v+N*(l_f + h*w)$, where $l_v$ and $l_f$ are the size of video- and frame-level annotations respectively. The final layer of the model predicts, for every $i$ in $\llbracket 1,L-1\rrbracket$, a vector $\widehat{o}_{i}$ (of size $q$, the number of possible tokens) which scores the likelihood of the $i+1$\textsuperscript{th} token given all preceding ones.

\paragraph{Training procedure.} To learn the parameters of $\mathcal{T}$, we load a complete sequence ($N$ frames corresponding to $L$ tokens), and try to predict the $L-1$ last tokens based on the $L-1$ first ones. This is done by maximizing the log-likelihood of the data using the cross-entropy loss:
\begin{equation}
    \mathcal{L}(\mathcal{T})=-\sum_{i=2}^{L}\log\left(\frac{\exp(\langle\,\widehat{o}_{i-1},e_{\scalebox{0.95}{$\uptau$}(i)}\rangle)}{\sum_{j}\exp(\langle\,\widehat{o}_{i-1},e_j\rangle)}\right),
\end{equation}
where $\scalebox{1.15}{$\uptau$}(i)$ is the $i$\textsuperscript{th} ground-truth token, and $e_j$ a vector of 0's with a $1$ in its $j$\textsuperscript{th} coordinate.

\paragraph{Inference.} During inference we use $\mathcal{T}$ to complete autoregressively an input sequence of tokens. To avoid known pitfalls (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, repetitive synthesis) and allow diverse outcomes, we use top-$k$ sampling whereby the next token is randomly chosen among the $k$ most likely ones, weighted by their scores $\widehat{o}$. Although $\mathcal{T}$ processes sequences of $N$ annotated frames, we predict ones of arbitrary length by using a temporal sliding window. Tokens corresponding to video- and frame-level annotations are given as input to $\mathcal{T}$ (not predicted, even for future frames) to guide the synthesis. We show in the experiments how this control can translates into a variety of interesting tasks.
\section{Related Work}
\paragraph{Video synthesis.} In its simplest form, videos are produced without prior information about their content. GAN-based approaches map Gaussian noise into a visually plausible succession of frames. For example, VGAN~\citep{vondrick2016generating} and ProgressiveVGAN~\citep{acharya2018towards}, adapt the traditional GAN framework~\citep{goodfellow2014generative} from image to video synthesis by simply using 3D instead of 2D convolutions. These approaches, including recent attempts such as G$^3$AN~\citep{wang2020disentangling}, are computationally expensive, and, by nature, restricted to synthesizing a fixed number of frames due to the constraints of their architecture. Other approaches predict latent motion vectors with a CNN~\citep{saito2017temporal}, or a recurrent neural network (RNN)~\citep{tulyakov2018mocogan, clark2019adversarial, saito2020train}, and generate frames with individual 2D operations. To avoid the shortcomings of information loss in the sequential processing of RNNs, we use an attention-based autoregressive model instead. Since we forecast temporal dynamics in a compressed space, a large temporal window can be used when predicting new frames, without having to resort to expensive 3D computations. Previous works have attempted to scale video synthesis to higher resolution by using progressive training~\citep{acharya2018towards}, sub-sampling~\citep{saito2017temporal, saito2020train}, reducing the dimension of the discriminator~\citep{kahembwe2020lower}, or redefining the task as finding a trajectory in the latent space of a pre-trained image generator~\citep{tian2021good}. Compression, together with efficient context-aware reconstruction allows us to synthesize videos at high resolution.

\paragraph{Controllable video synthesis.} Some of the aforementioned works~\citep{saito2017temporal, clark2019adversarial, saito2020train, wang2020disentangling} handle synthesis conditioned on a class label. Another popular control is to use a few priming frames to set off the generation process. This task, known as \textit{future video prediction}, has received a lot of attention recently. Methods based on variational autoencoders (VAE)~\citep{babaeizadeh2018stochastic, denton2018stochastic} have been proposed to account for the stochastic nature of future forecasting, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, the plurality of possible continuations, disregarded in deterministic predictive models~\citep{finn2016unsupervised, mathieu2016deep, vondrick2016anticipating}. Yet, their blurry predictions have motivated the incorporation of adversarial training~\citep{lee2018stochastic}, hierarchical architectures~\citep{castrejon2019improved, wu2021greedy}, fully latent temporal models~\citep{franceschi2020stochastic}, or normalizing flows~\citep{kumar2020video}. Another line of work infers spatial transformation parameters (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, optical flow), and predicts the future by warping past frames (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, grid sampling and interpolation as in~\citep{jaderberg2015spatial}) in RGB space~\citep{finn2016unsupervised, vondrick2017generating, hao2018controllable, gao2019disentangling, wu2020future} or in feature space~\citep{luc2020transformation}, typically using a refinement step to handle occlusions. Lately, autoregressive methods~\citep{weissenborn2020scaling, rakhimov2020latent} leveraging a self-attention mechanism~\citep{vaswani2017attention} have also been applied to this task. Our method benefits from the context-efficiency of approaches based on spatial transformation modules and the modeling power of autoregressive networks. In the meantime, other forms of control with interesting applications have emerged. Point-to-point generation~\citep{wang2019point}, a variant of future video prediction, specifies both start and end frames. State or action-conditioned synthesis~\citep{hao2018controllable, menapace2021playable} guides the frame-by-frame evolution with high-level commands. Additional works consider video synthesis based on one~\citep{pan2019video} or multiple layouts~\citep{wang2018video, mallya2020world}, another video~\citep{chan2019everybody}, or sound~\citep{vougioukas2018end, jamaludin2019you,  cherian2020sound}. To account for the variety of potential controls, we leverage the flexibility of transformers, and propose an unifying approach to tackle all of these tasks.
\section{Detailed architecture design}
\label{sec:architecture}
\setcounter{table}{0}
\renewcommand{\thetable}{A\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{A\arabic{figure}}

\begin{figure}[H]
    \centering
    \begin{tabular}{c@{}c}
        \multicolumn{1}{c}{\includegraphics[width=0.73\linewidth, valign=c]{neurips2021/figures/sup/auto-encoder-sup-2k.pdf}}
         & \includegraphics[width=0.21\linewidth, valign=c]{neurips2021/figures/sup/auto-encoder-sup-2bk.pdf} \\
         ~ \\ [-0.5em]
        \shortstack{(1) Conversion from image to feature space ($\mathcal{E}$) and vice versa ($\mathcal{D}$).\\ In the later case, context frames help producing new ones via $\mathcal{F}$.} & \shortstack{(2) Computations in $\mathcal{F}$ \\ at one resolution level.} \\
    \end{tabular}

    \hspace{0.2cm}

	\caption{In (1), illustration of the encoding ($\mathcal{E}$) and decoding ($\mathcal{D}$) architectures, and the interactions between the two thanks to the progressive optical flow and fusion mask estimation operator ($\mathcal{F}$). As opposed to $\mathcal{E}$, $\mathcal{D}$ processes features from low to high spatial resolution. To allow feature updates from context frame $x_c$ in the decoding phase of a latent embedding $z_s$, we first compute encoded features $e_c^k$ for all intermediate resolution levels $(r_k)_{k\in\llbracket 1, K\rrbracket}$ by applying $\mathcal{E}$ to $x_c$. We then decode $x_s=\mathcal{D}(z_s)$ from low ($k=1$) to high resolution ($k=K$) by iterating over the following steps: \textbf{(a)} apply one decoding sub-module to get $d_s^k$ from $d_s^{k-1}$, \textbf{(b)} use $d_s^k$ and $e_c^k$ (at corresponding levels) to estimate the flow $f_c^k$ from $e_c^k$ to $d_s^k$ and a fusion mask $m_c^k$, \textbf{(c)} use $f_c^k$ and $m_c^k$ to update $d_s^k$ with $e_c^k$ (see update rule), and \textbf{(d)} move to the next resolution level by going back to \textbf{(a)}. The residual computation of $f_c^k$ and $m_c^k$ from $e_c^k$, $d_s^k$, $f_c^{k-1}$ and $m_c^{k-1}$ is shown in (2) and detailed in the text.}
	\label{fig:skip-ae-sup}
\end{figure}

We describe the implementation choices for the different functional blocks which compose CCVS{}. Note that different parts of the framework may benefit from future advances in various research areas: autoregressive modeling, image synthesis, codebook learning and optical flow estimation for transformer $\mathcal{T}$, decoder $\mathcal{D}$, quantizer $\mathcal{Q}$ and flow module $\mathcal{F}$ respectively.

\paragraph{Encoder and decoder.}
Both the encoder $\mathcal{E}$ and the decoder $\mathcal{D}$, illustrated in Figure~\ref{fig:skip-ae-sup}-(1), use a sequence of $K$ modules handling information at different \revision{intermediate resolution levels $(r_k)_{k\in \llbracket 1;K \rrbracket}$} (with reverse ordering from $\mathcal{E}$ to $\mathcal{D}$). Each module is implemented by a residual block~\citep{he2016deep} which increases or decreases the spatial resolution by a factor $2$. In practice, we use the succession of two $3 \times 3$ 2D convolutions with LeakyReLU activations and a skip connection in the form of a $1 \times 1$ convolution.

\paragraph{Flow module.}
The flow module $\mathcal{F}$, also illustrated in Figure~\ref{fig:skip-ae-sup}-(1), allows the sharing of information between the encoding and decoding stages across different timesteps. Computations for one resolution level $r_k$ are detailed in Figure~\ref{fig:skip-ae-sup}-(2): The ``match'' operation computes the correlation between features in $d_s^k$ and its closest spatial neighbours in $e_c^k$. \revision{We thus obtain a 3D cost volume of size $N \times h_k \times w_k$, where $N$ is the number of neighbours considered and $(h_k \times w_k)=r_k$ is the spatial resolution.} We apply 2D convolutions to this cost volume to deduce the residual flow and mask. The ``refine'' operation is also a residual update of $f_c^k$ and $m_c^k$. It is made of 2D convolutions and takes as input the concatenation of features, mask and flow. Finally, the ``fuse'' operation update features from $\mathcal{D}$ with information from the context in $\mathcal{E}$ according to the update rule given in Figure~\ref{fig:skip-ae-sup}-(1) and detailed in Section~\ref{sec:ae}. For these operations, we take our inspiration from LiteFlowNet~\citep{hui2018liteflownet} which we adapt to handle not only the optical flow but also the proposed fusion mask. An extension to multi-frame context for this temporal skip connection mechanism is proposed in Appendix~\ref{sec:extension}.

\paragraph{Image and temporal discriminators.} The architecture of the image discriminator $\Delta_i$ is similar to that of StyleGAN2~\citep{karras2020analyzing}, and the temporal discriminator $\Delta_t$ is essentially the same, with 3D instead of 2D convolutions to account for the temporal dimension. To avoid the propagation of errors when decoding frames autoregressively during inference (see the ``shift-and-add'' module introduced for this purpose in Figure~\ref{fig:skip-ae}), we mimic this iterative process at train time: decode frames one by one, then encode them back to obtain context features for the next steps. The resulting synthetic videos are fed to $\Delta_t$. To avoid excessive memory consumption due to the recursive encoding / decoding operations, we only keep the gradients for the context features from the immediate preceding frame.

\paragraph{Quantizer.} The implementation of the quantizer $\mathcal{Q}$ is straightforward since it is just using a codebook of learnable embeddings. Yet, the non-differentiable nature of this operation prevents gradients from training losses (introduced in Section~\ref{sec:ae}) to backpropagate through $\mathcal{U}$ and $\mathcal{Q}$, and from $\mathcal{D}$ to $\mathcal{E}$ at the compression bottleneck. To solve this in practice, we use a straight-through gradient estimator~\citep{bengio2013estimating} which ignores the quantization step during backward-pass.

\paragraph{Transformer.} The transformer $\mathcal{T}$ is described in Section~\ref{sec:transformer}. Its input is obtained by transforming a sequence of tokens into a 2D matrix of size $L \times D$. For that, $\mathcal{T}$ associates with each possible token and each of the $L$ positions in the sequence a learnable embedding of size $D$.
To leverage the prior information that successive fixed-sized portions of the sequence represent annotated frames, we propose a spatio-temporal decomposition of the positional embeddings with a term indicating the timestep, and another one indicating the position within the frame-related portion. This is illustrated in Figure~\ref{fig:gpt}. The input matrix is computed by adding the token embedding and the positional embedding for each token of the sequence. $\mathcal{T}$ thus consists in the succession of multi-head self-attention layers, and position-wise fully connected layers.

\paragraph{Position estimator.}
In the experiments (Section~\ref{sec:experiments}), we mention that we manually annotated the $(x, y)$ location of the robotic arm (more precisely, its gripper) in 500 frames of the BAIR dataset. We use these to train a position estimator so that we can extract the trajectory of the arm from any new video (and use this trajectory to guide the state-conditioned synthesis of new videos). To facilitate training and accelerate inference, we design our position estimator to exploit latent features (obtained from images with $\mathcal{E}$) and to map them to the $(x,y)$ position of the arm. We use a simple architecture with a few downsampling convolutional layers and a fully connected layer to output the estimated position. The position estimator is trained by regressing the target 2D coordinates by minimizing the corresponding mean squared error.

\paragraph{Sound autoencoder.} To allow multimodal synthesis of video and sound, we first compute the short-time Fourier transform (STFT) of the raw audio to obtain a time-frequency representation of the sound. We then employ an encoding / quantization / decoding strategy similar to that used to construct $\mathcal{E}$, $\mathcal{Q}$ and $\mathcal{D}$ for video frames (without $\mathcal{F}$ in this case) to reduce dimension and obtain corresponding tokens for audio. These tokens are used as frame-level annotations to guide the synthesis, just as in state-conditioned synthesis.

\section{Role of the flow module}
\label{sec:role}
\setcounter{table}{0}
\renewcommand{\thetable}{B\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{B\arabic{figure}}
\setcounter{algocf}{0}
\renewcommand{\thealgocf}{B\arabic{algocf}}

\revision{Looking at Figure~\ref{fig:skip-ae-sup}, one may wonder whether the flow module $\mathcal{F}$ alone is sufficient for predicting features for future frames which would result in $z_s$ potentially being completely ignored. This would only apply if a frame was fully determined from the preceding one, yet, this is not the case due to the inherent stochastic nature of future prediction in the considered datasets. We have conducted a simple experiment to demonstrate that features $z_s$ are actually used and that they actively drive the dynamics of the scene: We have repeated the ``reconstruction'' experiment from Table~\ref{tab:ae-ablation} for which we compare the final model (last line) with an ablation with fixed $z_s$ (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, fixing $z_s=z_c$ for all predicted timesteps). Results are available in Table~\ref{tab:bair-fix-256}.}

\input{neurips2021/tables/sup/bair-fix-256}

\revision{We observe that the reconstruction performance is much poorer when we force $z_s$ to remain constant for all timesteps. Qualitatively, looking at the synthesized videos, we see that they do not exhibit any motion, as one would expect when fixing the encoded features. Moreover, thanks to the fusion masks estimated by the flow module $\mathcal{F}$ and used in Eq.~(\ref{eq:update}), it is possible to observe how much information comes from $z_s$ and from $\mathcal{F}$ respectively when generating $x_s$. Figure~\ref{fig:city-aug-256} shows some examples of estimated masks on Cityscapes. Those are white when the source is $z_s$, and black when it is context through $\mathcal{F}$. In practice, we see that they are white when the context is occluded and mostly grey in other regions. Thus, final videos are as much the result of context warping than direct decoding in non-occluded regions for this dataset. We observe similar behaviours on other datasets.}

\section{Flow inversion approximation}
\label{sec:inversion}
\setcounter{table}{0}
\renewcommand{\thetable}{C\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{C\arabic{figure}}
\setcounter{algocf}{0}
\renewcommand{\thealgocf}{C\arabic{algocf}}

For affine transformations, the exact inverse flow can be determined analytically.
This is not the case for elastic deformations for which we propose to approximate this inverse flow $f \in \mathbb{R}^{2 \times H \times W}$ from an input flow $g \in \mathbb{R}^{2 \times H \times W}$ (where $H \times W$ is the image spatial resolution).
Here, $g$ maps cells from a grid $\llbracket 1; H \rrbracket \times \llbracket 1; W \rrbracket$ to cells from the same grid (we ignore cells for which $g$ points out of the grid since they do not help in computing $f$). Such a mapping is not surjective, that is, all the cells in the grid are not necessarily reached. We compute the pixel-accurate inversion in cells for which there exists a direct mapping, and approximate others by iterative interpolation. That is, we keep track of inverted cells with a completion mask $c$ which we update with the closest cells in cardinal directions at every step.
For the elastic deformations that we use when training CCVS{}, around 80\% of the grid can be directly inverted in average, which is enough to accurately reconstruct the missing cells by interpolation.

\begin{algorithm}[H]
\SetAlgoLined
\KwData{Flow $g \in \mathbb{R}^{2 \times H \times W}$}
\KwResult{Approximation of inverted flow $f \in \mathbb{R}^{2 \times H \times W}$ }
\tcc{initialization of flow and cell-completion mask}
 $f\,=\textbf{0} \in \mathbb{R}^{2 \times H \times W}$\;
 $c=\textbf{0} \in \{0; 1\}^{H \times W}$\;
 \tcc{invert flow in cells for which there exists a direct mapping}
 \For{$(h, w) \in \llbracket 1; H \rrbracket \times \llbracket 1; W \rrbracket$}{
 $dh\,=g_{0,h,w}$\;
 $dw=g_{1,h,w}$\;
 $h'\,\,=\round(h+dh)$\;
 $w'\,=\round(w+dw)$\;
 \If{$(h', w') \in \llbracket 1; H \rrbracket \times \llbracket 1; W \rrbracket$}{
 $f_{0, h', w'}=-dh$\;
 $f_{1, h', w'}=-dw$\;
 $c_{h', w'}\,\,=1$\;
 }
 }
 \tcc{fill empty cells iteratively by interpolating neighbours' flow}
 \While{$\exists (h, w) \in \llbracket 1; H \rrbracket \times \llbracket 1; W \rrbracket \mid \, \sim{c_{h,w}}$}{
  $c'=\dilate(c) \land \sim{c}$\;
  $i\,\,\,\,\,=\blur(f)$\;
  $w\,\,\,=\blur(c)$\;
  \For{$(h, w) \in \llbracket 1; H \rrbracket \times \llbracket 1; W \rrbracket \mid  c'_{h,w}$}{
  $f_{0, h, w}=i_{0, h, w} / w_{h, w}$\;
  $f_{1, h, w}=i_{1, h, w} / w_{h, w}$\;
  }

  $c=c \lor c'$\;
 }
\caption{Flow inversion approximation.}
\label{alg:flow-invert}
\end{algorithm}
The blurring function $\blur:\mathbb{R}^{H \times W}\rightarrow\mathbb{R}^{H \times W}$ is a 2D convolution with $3\times3$ Gaussian kernel (and $0$-valued $1$-sized padding to preserve the spatial resolution), and interpolates flow values from neighbouring cells weighted by the their proximity; Dilation function $\dilate:\{0; 1\}^{H \times W}\rightarrow\{0; 1\}^{H \times W}$ propagates $1$-valued cells in a 2D boolean mask according to the cardinal directions (up, down, left, right); $\land$, $\lor$ and $\sim$ are notations which designate logical \textit{AND}, \textit{OR} and \textit{NOT} respectively.

\revision{We note that an alternative to flow inversion is to use the flow module $\mathcal{F}$ to compute the backward flow. This requires an extra forward pass in the flow module to compute both forward and backward flows and would result in additional GPU computations. Not to slow down training we prefer to run flow inversion on parallelized CPU processes as part of data loading. While both alternatives take approximately the same amount of time (0.15s on a GPU for backward flow estimation for a batch of 16 images at resolution $256 \times 256$, 0.12s on parallelized CPU for flow inversion for the same input) flow inversion is advantageous because, in our setup, GPU consumption is the limiting factor for speed (fully utilized GPUs, data loading on CPUs can run in the background).}

\section{Extension of the flow module to multi-frame context}
\label{sec:multi}
\setcounter{table}{0}
\renewcommand{\thetable}{D\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{D\arabic{figure}}
\label{sec:extension}

We extend the flow-module (introduced in Section~\ref{sec:ae}) which aims at reusing contextual information from single- to multi-frame context. We thus consider a context $(x_i)_{i \in \llbracket 1;c \rrbracket}$ consisting of $c$ frames. Just as before, for a given level $r_k$ and $i$ in $\llbracket 1;c \rrbracket$, we use $\mathcal{F}$ to compute the corresponding fusion mask $m_i^k$ and optical flow $f_i^k$ between intermediate encoded features $e_i^k$ (from context frame $x_i$) and the decoded features $d_s^k$ (that we wish to update). We recall that fusion masks $m_i^k$ handle occlusion by indicating for each spatial location the relevance of warped context features $e_i'^k=\warp(e_i^k, f_i^k)$, that is, whether features $e_i'^k$ correspond to $d_s^k$ at that location in terms of content, or not. Hence, we can derive from the fusion masks a confidence score $s_i^k$ defined as the context-wise normalization of features relevance, and use it to aggregate all the information that can be recovered from context by applying a weighted average:
\begin{minipage}{.5\linewidth}
\vspace{-0.2cm}
\begin{equation*}
    s_i^k=(\mathbf{1}-\sigma(m_i^k)) \oslash \left(\sum_{j=1}^{c}(\mathbf{1}-\sigma(m_j^k))\right),
\end{equation*}
\end{minipage}%
\begin{minipage}{.5\linewidth}
\begin{equation}
\begin{cases}
    m_{a}^k = \sum_{i=1}^{c}s_i^k \otimes m_i^k\\
    e_a'^k = \sum_{i=1}^{c}s_i^k \otimes e_i'^k
\end{cases}.
\end{equation}
\end{minipage}
where $\oslash$ represents the element-wise division. In other words, given a spatial location, $m_a^k$ is an estimate for the likelihood that the content of $d_s^k$ matches any of the $c$ context frames, and $e_a'^k$ is used to reconstruct this content faithfully by weighting proposals from the context by their relevance. This brings us back to the problem setup of Section~\ref{sec:ae}, so that the aggregated fusion mask $m_a^k$ and warped features $e_a'^k$ can be used to update $d_s^k$, just as in single-frame fusion.
The key benefits of this extension are the use of redundancy within context to better reconstruct the current frame, and the greater robustness to occlusion as different views are combined. This is demonstrated by the ablation study of the autoencoder in Table~\ref{tab:ae-ablation}.

\section{Additional results}
\label{sec:additional-results}
\setcounter{table}{0}
\renewcommand{\thetable}{D\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{D\arabic{figure}}

\input{neurips2021/tables/ucf101-unconditional-256}

\paragraph{Unconditional synthesis.} We apply CCVS{} to unconditional video synthesis (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, the production of new videos without prior information about their content) on UCF-101~\citep{khurram2012ucf101}, an action-recognition dataset of $101$ categories for a total of approximately $13$k video clips. Our approach leverages recent progress made in image synthesis by using StyleGAN2~\citep{karras2020analyzing} to produce the first frame. CCVS{} is able to improve over the state of the art on the FVD metric (computed on 2048 videos, under the same evaluation process as related approaches), but has an average performance ($24.47$) on the inception score (IS)~\citep{saito2020train} (computed on $10$k videos, again with the appropriate protocol). IS measures the coverage of the different categories, and whether one of these is clearly identifiable in each of the synthetic videos, through the lens of the Softmax scores of a C3D network~\citep{tran2015learning} trained for action recognition on the Sports-1M dataset~\citep{karpathy2014large} and fine-tuned on UCF-101. We also evaluate the performance of CCVS{} when considering a perfect image synthesizer (or one that has overfitted the training data) by using a real frame instead of a synthetic one to launch the prediction. This yields similar FVD but considerable improvements on IS ($41.37$). Note that CCVS{} favorably contribute to IS in this setting because videos made from the same real frames (by repeating them along the temporal axis) have much lower IS ($28.41$). A plausible explanation why MoCoGAN-HD~\citep{tian2021good} achieves a good IS score is due to it sampling multiple times from StyleGAN2~\citep{karras2020analyzing} (as opposed to once in our case, we use the same pretrained model). Indeed, MoCoGAN-HD synthesizes videos by finding a trajectory in the latent space of StyleGAN2. In this case, it may be easier to recognize a category because different modes can be interpolated (greater IS) but the outcome is less temporally realistic than ours (FVD near the one of still videos for MoCoGAN-HD). Future work will include trying to reconciliate both approaches.

\paragraph{Layout-conditioned synthesis.} We explore here layout-conditioned synthesis where one controls the semantic structure of new frames by attributing a class to each of their spatial locations. To this end, we apply CCVS{} on image and layout sequence pairs on the Cityscapes dataset~\citep{cordts2016cityscapes} which contains 3475 and 1525 30-frame sequences for train and test respectively. A single layout is annotated for each sequence, and we use a pretrained segmentation model~\citep{tao2020hierarchical} to obtain the remaining ones. Layouts are used as frame-level annotations for processing in $\mathcal{T}$. They are encoded, quantized, and decoded just as it is done for images. To save memory consumption during the decoding stage and ensure greater consistency between the two, frame and layout are decoded simultaneously with a single decoder $\mathcal{D}$ (we just supplement $\mathcal{D}$ with another prediction head to output layout masks). We show some qualitative results of CCVS{} on the synthesis of 30-frame videos given 3 starting frames and layouts for all timesteps in Figure~\ref{fig:city-layout-256}. Synthetic videos closely follow the target semantic structure, remain temporally consistent, and can handle complex motions. We repeat this synthesis process in Figure~\ref{fig:city-layout-off-256}, this time without layouts at subsequent timesteps. CCVS{} forecasts the semantic evolution of the scene on its own, predict corresponding layouts, and translate these into high-quality images. Both approaches (with or without layout at subsequent timesteps) are compared in Table~\ref{tab:city-layout-ablation-256}. Having this additional information allows to produce videos which are closer to the real ones. \revision{
A common architectural choice in layout-conditioned synthesis is to replace ResNet blocks~\citep{he2016deep} by SPADE blocks~\citep{park2019semantic} to strengthen the layout guidance in the decoding stage~\citep{mallya2020world}. Quantitatively, we see that it significantly improves the fidelity of the synthesized videos to the real ones in terms of SSIM and PSNR. Qualitatively, side-by-side reconstructions with and without SPADE in Figure~\ref{fig:city-layout-spade-256} show that the flow module allows temporally consistent textures (similar to the 3D world model used in~\citep{mallya2020world} but without being limited to static objects here) while SPADE enhances the compliance to the semantic structure, especially for small fast-moving objects.}

\input{neurips2021/tables/sup/city-layout-256}
\input{neurips2021/tables/sup/city-layout-off-256}
\input{neurips2021/tables/sup/city-layout-ablation-256}
\input{neurips2021/tables/sup/city-layout-spade-256}

\paragraph{Sound ablation study.} A simple ablation of CCVS{} on AudioSet Drum~\citep{gemmeke2017audio} is presented in Table~\ref{tab:drum-sound-ablation-64}. It shows that, as time goes by, the performance gap between unimodal and multimodal synthesis (when considering audio) increases. However, the improvement on account of the audio information seems relatively small compared to the overall performance. We suspect this is due to a minor part of the scene being animated (the drummer upper body) and to relatively fast motion (it is difficult to strictly minimize the image-to-image error between real and synthetic videos in this case).

\input{neurips2021/tables/sup/drum-audio-ablation-64}

\paragraph{Point-to-point synthesis.} We explore further the performance of CCVS{} on the point-to-point synthesis task which we introduced in Section~\ref{sec:experiments} of the article. We show some quantitative results against different baselines in Table~\ref{tab:bair-p2p-64}. CCVS{} is on par or significantly outperforms the state of the art on $3$ out of $4$ metrics. It produces more faithful reconstructions when latent features are known. CCVS{} is more likely to synthesize the true outcome among $100$ predicted samples compared to baselines. It has a good control point consistency (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, the last synthetic frame is close to the corresponding ground-truth), but lower diversity than prior arts. We note, however, that SSIM diversity may capture the capacity of producing different trajectories between start and end points, but may also increase due to visual quality degradations. Like previous works~\citep{wang2019point}, CCVS{} is able to produce videos of arbitrary length between the target start and end frames. We recall that the end frame is used as a video-level annotation in the transformer model $\mathcal{T}$ (Figure~\ref{fig:gpt}) so that it is taken into account when predicting frames at earlier timesteps. To allow any timestep for the end frame (possibly greater than the capacity of $\mathcal{T}$), we make its positional embedding reflect the gap with current predicted frames. Namely, the positional embedding of the end frame is updated as the sliding window moves forward in time and gets closer to the end timestep. We show qualitative results for different video lengths in Figure~\ref{fig:bair-p2p-length-256}.

\input{neurips2021/tables/sup/bair-p2p-64}
\input{neurips2021/tables/sup/bair-p2p-length-256}

\paragraph{Class-conditioned synthesis.} Next, we conduct a qualitative study of video synthesis conditioned on a class label on the Weizmann dataset~\citep{gorelick2007actions}, which consists of short clips captured from a fixed viewpoint from $9$ subjects, each performing the same $10$ actions. We use $8$ subjects for training and leave $1$ out for testing. CCVS{} tackles this task by specifying the class label as a video-level annotation in $\mathcal{T}$ (Figure~\ref{fig:gpt}). We can make the held-out subject perform the learned actions accurately given a single input image (Table~\ref{fig:weizmann-class-128}). Or we can synthesize new videos without any prior knowledge other than the desired class label (Table~\ref{fig:weizmann-class-nof-128}).

\input{neurips2021/tables/sup/weizmann-class-128}
\input{neurips2021/tables/sup/weizmann-class-nof-128}

\paragraph{Qualitative ablation.} We repeat the ablation study of the autoencoder conducted in Table~\ref{tab:ae-ablation} on BAIR~\citep{ebert2017self}, by giving some qualitative samples of the reconstruction this time. The reconstruction results of 30-frame videos given the first frame and the compressed features at subsequent timesteps can be found in Table~\ref{tab:bair-ae-qualitative-ablation-256}. To best compare different ablations of CCVS{}, we use the same input data for all. A model trained with the $L_1$ loss in RGB space reconstructs the static background and the robotic arm well, while other objects look very blurry. When using the same loss in the feature space of a $\VGG$ net~\citep{simonyan2015very}, objects display higher-frequency details, but look transparent. Having the image discriminator $\Delta_i$ allows much more realistic outcomes, although these are still not temporally consistent, nor do they faithfully correspond to the ground-truth video. Having the temporal discriminator $\Delta_t$ seems to improve over the temporal consistency issue. Thanks to the flow module $\mathcal{F}$, reconstructed videos are closer to the ground-truth ones, in particular at early timesteps. However, we see that flow imperfections cause synthesis artefacts which add up as time goes by, and result in quality degradation. The self-supervised technique used to learn $\mathcal{F}$ mitigates this shortcoming. Finally, larger context windows and longer training times help to better recover details (see the more realistic texture and shape of manipulated objects for instance). The final proposal creates high-quality videos almost indistinguishable from real ones, and shows true interactions with objects (a ball and a colorful gun in this example). We see that in case of disocclusion (around $t=20$ the upper right corner is revealed), our model is able to synthesize plausible arrangements of objects despite lossy compression. These new objects remain consistent as the synthesis process unfolds (see $t=30$) thanks to the autoregressive nature of our approach.
\input{neurips2021/tables/sup/bair-ae-qualitative-ablation-256}

\paragraph{Long-term synthesis.} \revision{We assess the ability of CCVS{} to generalize to long-term prediction through a qualitative study. We compare, for a model trained on sequences of length 16, the synthesis of 90-frame videos conditioned on a single frame and using different context windows for the flow module: 0 (not using the flow), 1, 4 and 16. Results on BAIR, Kinetics and AudioSet-Drums are available in Figure~\ref{fig:long-term}.  When the flow is not used (size 0, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, a naïve extension of~\citep{esser2020taming} or a setup close to~\citep{rakhimov2020latent}) videos are quite unstable and we see colors shifting over time. With a single-frame context window (size 1), there is a greater temporal consistency (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, the drum sticks are better preserved on AudioSet-Drums, the semantic structure shows higher fidelity to the input frame on BAIR or Kinetics), but in the long-term, synthesis artefacts add up due to the autoregressive decoding which leads to visual deterioration and saturation. These effects stand out the most on BAIR and Kinetics. The proposed multi-frame context extension (Appendix~\ref{sec:extension}) reduces this issue (size 4), and the latter becomes barely perceptible with an even larger context window (size 16). We found that even small context windows on AudioSet-Drums are sufficient thanks to the high inter-frame similarity.}
\input{neurips2021/tables/sup/long-term}

\section{Augmentation strategy}
\label{sec:augmentation}
\setcounter{table}{0}
\renewcommand{\thetable}{F\arabic{table}}
\setcounter{figure}{0}
\renewcommand{\thefigure}{F\arabic{figure}}

As discussed in Section~\ref{sec:ae}, the self-supervision process used to train $\mathcal{F}$ reconstructs a static image $x$ from both its latent representation and an augmented view as context by estimating the flow and mask corresponding to that augmentation. We illustrate this process on the Cityscapes dataset in Figure~\ref{fig:city-aug-256}. We recall that, in this case, the context image can be writen as $x_c=o_c \otimes \blur(\aug(x))$, where $o_c$ is an occlusion mask, and $\blur$ and $\aug$ some blurring and augmentation (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, spatial transformation) functions respectively. We can invert the flow corresponding to transformation $\aug$ thanks to Algorithm~\ref{alg:flow-invert}. The output corresponds to the target ``Real Flow'' in the illustration. As for the target ``Real Mask'', it is obtained by warping the occlusion mask with the inverted flow. Looking at synthetic reconstructions, we see that visible details in the context are well recovered despite the augmentation. Indeed, real and synthetic flows match in non occluded areas (indicated by the dashed red line). Moreover, adversarial training produces high-quality images from blurred and occluded contexts. We notice that textures do not always match in real and synthetic images (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, building windows, car appearance). Yet, thanks to recovery losses, the semantic structure of the scene is preserved. Finally, we see that darker and lighter areas of the synthetic masks correspond to the same areas in the real ones. The darker the synthetic masks, the more features are updated from context. The \textit{whitish} part indicates that in occluded areas the final image is reconstructed from its latent representation alone, and the \textit{greyish} part shows that for other regions context is effectively reused (and latent representations still contribute to the final outcome). It is interesting to see that edges are visible in the synthetic masks. This indicates that those regions may be easier to detect (darker color) and play an important role in the image alignment process (estimation of the flow).

\input{neurips2021/tables/sup/city-aug-256}

redpajama_set_name: RedPajamaArXiv


Samsung has officially released Galaxy S9, the first smartphone that lets you change the aperture on its camera. While it has several improvements over the previous generation of Samsung phones, it also comes with a couple of new and interesting features regarding the camera capabilities, such as the super-fast 960fps slow-motion.
Along with the Galaxy S9, there's also the larger Galaxy S9+, which has a dual camera. Similar to the previous model Galaxy S8, the S9 also comes with Dual Pixel system. However, the new model lets you physically change aperture. It switches between f/1.5 and f/2.4, which is the first time a variable aperture is fitted into a smartphone.
Maxime Guirauton, marketing director for Samsung, says that the company was trying to "copy what the human eye can do automatically." He adds that the new dual-aperture system combines "the best of both worlds": it lets you shoot in the dark, but without the outdoor photography in good light suffering. As for the low-light shooting, the new Samsung Galaxy S9 shoots 12 photos in three groups of four and then combines them to reduce noise. So, I guess with this phone you won't have to do this in Photoshop.
Another novelty is the super slow-motion function that lets you shoot at 960 fps. It's also equipped with motion-detection, so it will start recording as soon as the movement is detected.
If you're interested in some silly camera-related features, you might also be interested in animojis that Samsung Galaxy S9 lets you make. Just like its greatest rival Apple, Samsung now also lets you make animated emojis with your facial expression using the facial recognition technology.
Here are a couple of hands-on videos so you can see the Galaxy S9 in action.
Samsung Galaxy S9 will be ready for preorders on 2 March, and it starts shipping on 16 March current year. It will be available in three colors: black, coral blue and lilac purple. The regular model costs $719.99, and the Plus model will cost you $839.99.
hmm wait … my Huawei Mate 9 allows me that too – do they do it digitally with Huawei? I understand it can be possible with lower f-numbers but with high f-number and more depth of field, dunno?
This is NOT the first smartphone to include variable aperture. You'd think you'd check your facts before stating that in an article. I know for a fact the Nokia N86 in 2009 had variable aperture (not sure it was first).
Now don't you sound stupid, that phone he listed is classified as a smart phone. Do your research.
Haha wow. Look up the definition of a smartphone before you go and say something foolish like that.


Well, the beans have officially spilled and even typing this out still seems a little unreal. We're expecting! A baby.. Our baby… A little Stiemsma! This entire process is hands down the most amazing thing in the world and we are getting to experience it, how blessed are we!?
He knew right away what that meant.
I think both Greg and I wanted to jump up and down and cry tears of happiness, but the truth is- neither of us reacted that way. We were incredibly filled with joy but the shock and surprise (in a good way) was so overwhelming we almost didn't have words or actions to express how we felt. Both of us quickly found ourselves apologizing for lack of emotions, but it was mutual and we were on the same page with things.
Over the next few weeks reality slowly set it, and even into today, we just can't believe that we will be bringing a child into this world!
Well, my mind is turning and I know Greg's is too. The idea of being parents to us is an incredible journey (rewarding and challenging) that we're ready for. I've done lots of reading and pinning (if you follow me on Pinterest I'm sure you already knew I was pregnant, lol). Planning for the nursery and hunting for tips/tricks/hacks on pregnancy, and a few "what to expect" books… Truth is, I don't know what to look for or buy, but I know we've got lots of time.
More than anything I am so impressed with God. Look at what he made possible! Look at what the woman's body can do – a process that all mothers share yet it is so incredibly different for us all and I love that even more about it.
When I was a kid myself, I never imagined what my wedding would be like but I did always imagine what being pregnant would be like. I thought I would be the lady that put headphones on their belly because it supposedly made the child more intelligent. Hey, I never said I still wasn't going to do that, but the belly isn't quite big enough yet. This pregnancy journey is exciting and unknown for sure but we are taking one day at a time and feeling so excited for this change in our lives. And, we are thrilled to finally share the news with the rest of the world!
Other than an expanding family, Greg and I both have some exciting career changes.
After about a year and a half with my current company, I had an opportunity with a new company unravel. Like most good things, it wasn't something I was looking for. I had helped out with a small basketball video gig at this new company and found myself incredibly passionate about their mission and products as well as fitting in great with their team.
After a few conversations and a lot of thinking on my end, I accepted a full-time position with Dr. Dish, a basketball training company that specializes in shooting equipment. I officially joined their marketing team this week and couldn't be happier about it. Seems like a pretty good fit, would you agree?
Now for Greg, I need to brag a little bit about him. Once Greg was out of the NBA our relationship definitely had its challenges, something that comes with most transitioning times. We were used to long distance and space, and now we were together 24/7. I started working full time and I wanted Greg to hop directly into another career and discover his passions. Over time, my patience grew and I learned this transition wasn't going to happen over night. I still joke that he is a "stay at home dog dad", which, he still is. I never meant that as an insulting thing. Anyone who knows Greg knows he is kind, loving, caring, patient, and selfless. I love him at home, but part of me still wanted more for him. He needs to touch more people with his grace & talents!
Now, two years have passed and Greg is finding his footings in a second career. He took the initiative to take real estate classes and do some basketball training on the side and I couldn't be more excited to see doors open for him! Oh, and did I mention he is now an official Keller Williams Realtor? Most of you know that we have some rental experience, and with Greg being a great people-person, the fit is stellar.
We don't know exactly what life will look like after baby arrives due to some of these changes, but we are not worried about finding a balance for Mom, Dad and Baby. Whoops- and Diezel of course!
More pregnancy updates to come.
*Photos taken by photographer Katy Kalleberg- see her gorgeous IG feed!
Many thanks Natalie!! I'm sure Diezel will want some extra love from you in the month of December 🙂 We love you too!


Update 11/23/2005: Months later, this is still apparently a popular topic, as it's one of the most viewed posts here. Time has passed though, and there's a better option: RadRails. It's built on Eclipse, but includes Rails-specific tooling, which the configuration discussed below does not. It's well worth your time to check out.
In this short tutorial, I'm going to walk you through setting up a code editing environment for Ruby on Rails using the Eclipse IDE.
Eclipse isn't really an IDE in and of itself. It's more like a platform to build IDE's on. It was developed by IBM and then released as an open source project. The code is now owned by the Eclipse Foundation. IBM uses Eclipse as the foundation for all of their development tools, such as Rational Application Developer, their J2EE development tool. There are a variety of other companies and open source projects developing plugins as well as full-blown IDE's using Eclipse. See the Eclipse website for further information.
Note: Eclipse is a pig when it comes to memory and processor (mostly memory). If you don't have a 1.5ghz+ machine with 1gb of ram, you may find using it a little frustrating. I am using it on a machine with a 1.1ghz P4 and 512mb of ram, and it does OK, but it's not what I would call quick.
What we are going to do here is install Eclipse (if you don't already have it), and then add two plugins to it. The first plugin will add support for editing Ruby code. The second will add support for HTML editing. Then we'll make some configuration changes to make everything work nicely, and that will be it.
NOTE:This is not a tutorial on Eclipse. I'm not planning on covering the ins and outs of Eclipse here, there are plenty of other sites that offer that kind of information. If you're interested in seeing a more in-depth tutorial on using Eclipse to edit Ruby/Rails code, drop me a line at \[larry AT approachingnormal.com\], and I'll consider developing one.
The first thing you need to do is install Eclipse, if you don't have it already. Head over to http://www.eclipse.org, then download and install the latest version for your platform. Make sure to pick a stable release and not one of the milestone releases. Once you have it installed, start up Eclipse and go on to the next step.
Once you are in Eclipse go to Help ~~> Software Updates~~> Find and Install. This brings up a wizard-style dialog. On the first screen, make sure that "Search for new features to install" is selected, and click Next. The next screen let's you add/edit update sites where eclipse will look for additional functionality. We're going to add two sites here. First, click on the "New Remote Site" button. When the New Update Site dialog pops up, enter "Ruby" into the Name field, and http://rubyeclipse.sf.net/updatesite in the URL field. Then click OK. You will now see your Ruby entry in the list of update sites. Click on the checkmark to the left of the entry to include it in the list of sites to search.
Now, click on the "New Remote Site" button again. Fill in the values as before, this time using "Web" as the Name, and http://pipestonegroup.com/eclipse/updates as the URL. Click OK as before, and make sure you check the checkbox to the left of the Web entry. When you're all done, click Next.
The next screen takes you to the Search Results screen. This will give you a list of features you can install. We're interested in two here. Select the checkboxes for the latest version of "Ruby Development Tools" (currently 0.5), and "net.sourceforge.phpeclipse" (currently 1.1.4). Click Next. On the next screen you'll accept the licenses, and click Next again. On the final page, you'll choose the location to store the plugins. Select the default unless you have a reason to do otherwise. I had to add a new site to store mine in my home directory on Linux due to not having permissions to the default location. On a Windows machine you should be fine with the default. Now click Finish and confirm the dialogs that pop up. The download process may take a little time depending on your internet connection.
We need to let Eclipse know what to do with rhtml files. By default, Eclipse will treat it as a text file, which isn't what we want. In Eclipse, go to Window ~~> Preferences. In the tree on the left, navigate to Workbench~~> File Associations. Now click the Add button at the top of the dialog, next to the File Types list. Enter \*.rhtml in the dialog that appears and click OK. Now select our new type in the File Types list and then click the Add button at the bottom of the screen next to the "Associated Editors" list. Choose "HTML Editor" from the list that appears, and click OK. Now close the Preferences screen.
Now click Finish. Once the source is imported, you should see be able to navigate through the project tree in the Navigator window. If you click on ruby and rhtml files, you will see that they are opened in their proper editors.
Eclipse doesn't really understand ERB files. It would be great if the Ruby plugin understood them, as it would make the syntax highlighting much better. It would also allow for code completion, etc.
WEBRick integration. It would be nice to be able to stop/start WEBRick from within the IDE.
Perhaps someone will tackle this project, as it seems there's a good foundation already.


text:
Filme | Seriale | Anime
The Skull Man: Complete Collection 2007 DVD
MVM Entertainment
All 13 episodes of the Japanese anime based on the manga series created by Shotaro Ishinomori. The show follows 23-year-old journalist Hayato Mikogami (voice of Makoto Yasumura) as he returns to Otomo City to investigate reports of a murder committed by the mysterious Skull Man (Hiroshi Tsuchida). The episodes are: 'City of the Dancing Mask', 'The Man from the Past', 'Crimson Rain Falls in the Afternoon', 'The Ghost That Goes for a Stroll', 'The False Blind Spot', 'Banquet of the Grim Reaper', 'Master of the Beasts', 'The Maiden Whispers in the Moonlight', 'Blessed Are the Impure', 'The Circus Laughs at Midnight', 'Darkness Take My Hand', 'Bloody Eve' and 'Black Fable: All That I Love Leaves Me'.