text:
We've Reached a Horrifying New Stage
Trump's crowds will follow him anywhere. How dark will the path get?
Donald J. Trump, American president, went down to the state of Tennessee Tuesday night and constructed a whole world for his supporters to fear, to hate, and possibly, to lash out against. The president was there to help defend the Senate seat Republican Bob Corker is vacating, because Representative Marsha Blackburn, a bona fide loon, is trailing Democrat Phil Bredesen in the polls. This, despite the fact that Trump won the solid-red state by 26 points in 2016. Trump's rhetoric seemed to reflect this state of affairs, and a more generalized anxiety about the 2018 midterms.
In short, the President of the United States suggested that swathes of the nation he leads have been overrun by gangs of criminal immigrants, and that his political opponents expressly support these gangs.
TRUMP: "What was the name [I used to call MS-13 gang members]?
CROWD: ANIMALS!
Trump has unleashed the ugly in America.pic.twitter.com/Fg5yJQ3Q7p
— SFDB (@sfdb) May 30, 2018
"Blood-stained killing fields."
"Savagely burning, raping, and mutilating."
This was a return to Trump's Inauguration Day vision of American Carnage. It was a reminder to his supporters, who might feel less motivated to go to the polls in November, that there are a great many things to be afraid of—real or, in this case, summoned from the dark recesses of the presidential mind.
There was also the suggestion Democratic leaders, including Nancy Pelosi, have demanded "respect" for criminal gang members—a predictable mischaracterization of what she said, according to TPM:
Pelosi did not endorse the gang, but took aim at Trump's rhetoric, saying, "Calling people animals is not a good thing." She added that "we all have to recognize that as we respect the dignity and worth of every person."
The implication, of course, is that Democrats somehow support this criminal gang, because they oppose the leader of the world's most powerful nation calling people "animals," or they oppose The Wall. Trump sought to make the connection unequivocal:
Trump calls Tennessee senatorial candidate Phil Bredesen "an absolute, total tool of Chuck Schumer, and of course, the MS-13 lover Nancy Pelosi," at a rally in Nashville #tictocnews pic.twitter.com/GkptwOp7cb
— Bloomberg Quicktake (@Quicktake) May 30, 2018
"The MS-13 lover." This is, obviously, an egregious lie. But the intent is also obvious: to suggest Democrats are not merely political opponents, but are actively endangering the safety of you and your family. They are prioritizing their politics over the good of the nation. They are enemies of the state.
But perhaps the most unnerving portion was the call and response, where the president's supporters dutifully followed him down the road of calling other human beings "animals." They did so gleefully, as they once engaged in back-and-forths about The Wall and how Mexico Is Going to Pay For It. (The fact that there is still no wall, and that Mexico has steadfastly refused to pay for it, never seems to matter. Instead, Trump lied and said construction of The Wall has begun.) It was a sign that the faithful are taking to the new tactics with a dark enthusiasm.
We have discussed, ad nauseam, how it is a waste of time to get in the mud with this president on whether he originally referred to just MS-13 members as "animals," or whether he was referring to some larger group of undocumented immigrants:
Trump constantly conflates all immigrants with violent criminals, and never bothers to clarify that most in this group have committed no crime on American soil and are simply seeking honest work and a better life. (In fact, the undocumented are less likely to commit crimes than native-born Americans.) He never makes the distinction, because to him, there is no difference worth remarking on—or worse, he finds it politically fruitful to ignore it.
This non-distinction is also reflected in his policies, as Vox outlined:
Arrests of immigrants without criminal records have also spiked. During President Obama's last year, about 16 percent of ICE arrests were of noncriminal immigrants; each month since July 2017, between 32 and 40 percent of arrestees have been noncriminals."
This is also reflected in the fact that ICE has been caught labeling people as MS-13 members with zero justification, so that they are refused bail and can more easily be deported. It's almost like the main priority is getting people—most of whom are of a certain complexion—out.
Also, the president was caught last week as having said this in the Oval Office:
Trump reminded them the crowds loved his rhetoric on immigrants along the campaign trail. Acting as if he was at a rally, he then read aloud a few made up Hispanic names and described potential crimes they could have committed, like rape or murder. Then, he said, the crowds would roar when the criminals were thrown out of the country — as they did when he highlighted crimes by illegal immigrants at his rallies, according to a person present for the exchange and another briefed on it later. Miller and Kushner laughed.
It is painfully obvious that this president has no problem singling out the very worst among undocumented immigrants and holding them up as representative of the group. He wants MS-13, and Kate Steinle's killer, and all the other worst elements to be the face of the undocumented population. It's all he talks about, until the only image that appears in his supporters' minds when they hear the term "illegal immigrant" is someone of a certain complexion who has committed a violent crime. Does it still seem worth debating whom, exactly, Trump is calling an "animal"?
Perhaps, in the short term, he's merely hoping to boost Republican midterm turnout through the raw power of fear. The risk, however, is that this spills into the kind of fervor that leads people to do terrible things—things they might hesitate to do to a person, but not to an animal.
What happens if this powder keg's fuse is lit by some extraordinary event? What happens if Democrats, whom the president has told his supporters are backing violent immigrant gangs, win big in November? Will the same supporters we see chanting so lustily in the crowd simply accept the results of this democratic process? Constructing a world of American Carnage can be a dangerous game.
You Gave Trump the Benefit of the Doubt?
Lindsey Graham Should Really Just Shut Up
Windows Weren't the Only Things Broken in Congress
Cities Planning to Resist Trump's Immigration Plan
At Least We Got to See Trump Get Beat Like a Drum
Turns Out People Still Buy Trump's Bullsh*t
There Was Only One Really New Lie
The Un-American ICE
Another Sign of How Deep Trumpist Poison Reaches

redpajama_set_name: RedPajamaCommonCrawl


text:
Other Games / By zoe
Ace Combat 6 Game Review:
Ace Combat 6: Fires of Liberation is one of the best flight simulators I have ever come across and I really enjoyed the experience of playing it. The game has top of the range graphics and a really smooth game play engine to boot which make the game easy to play and follow. The story line is just as good and has a large amount of detail within it and I love it to pieces.
We have also compiled a Review System for this, of which Ace Combat 6 is Rated as Four Stars, the second highest possible score available from the Game Review Team.
The Website address for Namco Bandai is: http://www.uk.namcobandaigames.eu
The Website address for Ace Combat 6 is: http://www.uk.namcobandaigames.eu/product/ace-combat-6-fires-of-liberation/xbox-360


text:
Lupe fiasco tetsuo and youth
23.04.2020 23.04.2020 Tajar
17 rows · Find album reviews, stream songs, credits and award information for Tetsuo & Youth - . Jan 20, · About "Tetsuo & Youth" Tetsuo & Youth is the fifth studio album by Lupe Fiasco, released on January 20, through the Atlantic Recording Corporation. Make sure . DETAILED EXPLANATION OF LUPE FIASCO'S TETSUO & YOUTH Lyrics: -- After listening to Lupe's new album, Tetsuo & Youth, multiple times, I specifically want to begin this analysis by getting into.
17 rows · Find album reviews, stream songs, credits and award information for Tetsuo & Youth - . DETAILED EXPLANATION OF LUPE FIASCO'S TETSUO & YOUTH Lyrics: -- After listening to Lupe's new album, Tetsuo & Youth, multiple times, I specifically want to begin this analysis by getting into. Jan 22, · Lupe Fiasco's fifth album, Tetsuo & Youth, is the most focused, thoughtful, and satisfying project he's offered since 's The riverwalkathof.comer set Author: Jayson Greene. Jan 20, · About "Tetsuo & Youth" Tetsuo & Youth is the fifth studio album by Lupe Fiasco, released on January 20, through the Atlantic Recording Corporation. Make sure .View credits, reviews, tracks and shop for the CD release of Tetsuo & Youth on Discogs. There is a scene in Charlie Aheam's film Wild Style where a gallerist by the name of Niva Kislac invites some rappers and a man named. We and our partners use cookies to personalize your experience, to show you ads based on your interests, and for measurement and analytics purposes. Tetsuo & Youth is the fifth studio album by American rapper Lupe Fiasco. The album was released on January 20, , by 1st & 15th Entertainment and Atlantic. Lupe Fiasco's fifth album, Tetsuo & Youth, is the most focused, thoughtful, and satisfying project he's offered since 's The Cool. Whatever set the stage for him, he dives back into rapping for its own sake like it's a big dusty novel he's been waiting to pick back up for. Metacritic Music Reviews, Tetsuo & Youth by Lupe Fiasco, The fifth studio release for the rapper features guest appearances from Ab-Soul and. This is Lupe Fiasco's 5th studio album w Atlantic Records. on this album the listener will be delighted to hear Lupe's return to the sharp lyricism and soulful. Here's The "Funky Drummer" Sampled By Dr. Dre & Kanye West. Regina Hall Takes The Lil Rappers Quiz | The Genius Test. Juice WRLD. Album rollouts don't come much odder than Lupe Fiasco's Tetsuo & Youth. Since the Chicago-emcee announced the project during the see more, oboe concerto er scribd vaughan williams,lagu burning spear black,source,https://riverwalkathof.com/brc-global-standard-for-consumer-products.php
see the video Lupe fiasco tetsuo and youth
Tetsuo & Youth First Reaction, time: 22:00
Tags: Benzemez kimse sana games, Nana mouskouri early one morning skype, Wordpress socio multi vendor plugin, Materials and their properties pdf, Steve fuzz creepstep firefox
2 thoughts on "Lupe fiasco tetsuo and youth"
For a long time searched for such answer
Gumuro says:


Comprehensive list of Oceanside, New York based companies offering installation, maintenance and repair of home and business security systems, CCTV & burglar alarms, guard and video surveillance services.
Digital Automated Security. opened its doors for business in 1990. Our aim has always been to be a complete service alarm firm in the Northeast area. We appreciate your interest and look ahead to gaining your business. At Digital Automated Security., we do our job as though our lives depend on it, because everyday they do.
Any firm can sell security products. Sales staff are essential to any firm. But to you, the client, of far greater consequence is a firm's operation. Our operations staff is the finest in the business. Eye On You believes that every project, no matter how large or how tiny, requires the same superior level of performance in equipment and installation.
Established in 1980 by Michael Voldo, and joined by Anthony Mileo in 1999, DEFENSIVE DESIGN SECURITY SYSTEMS, Ltd. is backed with over forty years of combined industry experience. With our major office situated at 200 West Sunrise Hwy., Freeport, Defensive Design Installs and services systems in the Greater New York Area.
Throughout Long Island, families know the name that is synonymous with best quality comfort and steadfast dedication - Allyn Services. For sixty years, Long Islanders have called on Allyn for premier heating, fuel efficiency analysis and alarm services. We've gladly answered that call, and we are pleased to be the most trusted name in home comfort.
Esquire has been incorporated as a New York State licensed and insured private investigation business since 1989, and has offered airport security services - with Port Authority credentials - for more than a decade. Esquire's work at JFK and EWR involves real-time supervision and shadowing of all manner of valuable cargo in and around the airport as well as while it's in-transit to and from airports.
RBK NEW YORK CORP is a service and installation firm that integrates your lighting control, audio/video distribution, intercom & telephone system, surveillance cameras, HVAC control, electronic drapers, shades or blinds opener into one intelligent system. This is what is recognised as Home Automation, that your house intelligently serves you in a way that always makes you feel welcome at home.
Finest Systems, known as Finest Security, is a complete service security and communications product and installation firm, providing the finest products and services within the NE. In business over fifteen years, we supply courteous and professional service to huge, tiny and medium businesses, as well as to the personal consumer.
Since 1981 Protection Plus Alarm Service. has been protecting homes and businesses on Long Island including The North Fork, NYC Five Boroughs, and Westchester. We are a local independant alarm firm, so we supply you with personalized service. Large enough to supply quality service, but tiny enough to cope with our clients on a individual level.


A hive at Nottingham Trent University's city campus is now home to 20,000 bees.
The hive, which is located on the green roof of the Newton Building, was installed in June last year.
A camera has been placed in the hive so visitors to the university's website can watch activity.
The university has bees on all three of its campuses.
Brackenhurst,near Southwell, hosts hives for the Nottinghamshire Bee Keepers' Association, while at Clifton bees are kept for research.


For many people, the sight of deer in their backyard on a brisk autumn morning is a wonderful start to the day. But, as we continue to encroach into formerly wild areas, are we putting ourselves and our pets at risk?
As we build more homes in formerly rural areas, contact with wild animals increases. Much of this new interaction has unfortunate consequences for the wildlife. This is evident by the number of dead skunks, raccoons, and possums along the roadside.
Humans and their pets are also in danger in these wild interactions. Along with deer come blood-thirsty ticks and an array of bacterial diseases. Raccoons and skunks bring the terror of rabies to our backyard and even the humble mouse has the potential for spreading deadly Hantavirus.
Skunks and raccoons are two important reservoirs of rabies in North America. Prior to 1977, rabies was very rare along the mid-Atlantic states and New England area. But, a human managed relocation of raccoons from Florida to West Virginia in the late 1970s has unleashed a new epidemic of rabies in these areas.
Rabid raccoons often become nice and approachable and many people are tempted to take the animal into their yards or homes. Skunks, on the other hand, will become overly aggressive and actively attack humans and pets.
Raccoons also harbor a significant parasite known as the raccoon roundworm or Baylisascaris. These large worms are associated with severe or even fatal central nervous system disease in many mammals. The eggs are passed in the feces of the raccoon and then encountered by other animals, including children. The parasite can also mature in our dogs. This means that it is possible our pets are helping to contaminate larger areas with this potentially fatal worm.
They may be small, but many mice and rats can carry a killer virus. First discovered in the Four Corners region of the US, Hantavirus (or Sin Nombre virus) is now found in more than 30 states. Because of a long incubation period (one to five weeks), many people are unaware of a problem until too late. Thirty percent of affected individuals die. This disease is spread through rodent droppings, urine and saliva. It is possible to become infected after cleaning a house or barn where rodents have been in residence. Thankfully, our pets are not affected by this virus.
According to the Centers for Disease Control (www.cdc.gov), the white tailed deer population in North America is now approaching levels not seen in more than 200 years. Although beautiful to look at, many wild deer carry some unwanted passengers, like ticks.
Ticks are the primary vector for several serious bacterial diseases like Lyme Disease, Rocky Mountain Spotted Fever and Tularemia. As we have built new subdivisions in rural areas or reforested old agricultural lands, these diseases have shown significant increases, both in humans and our dogs.
Finally, a single celled organism known as Giardia is the most common intestinal parasite of people in North America. In fact, people comprise the main reservoir of this disease. But, wild animals, like beavers, muskrats and small rodents also make up an important additional source of infection. This parasite can cause severe diarrhea, anorexia, and weight loss in both people and pets.
Avoiding contact with wildlife is the number one rule. Not only will it help prevent disease transmission, but it will also stop traumatic injuries from fights or chases through the woods.
Avoid the temptation to feed the local wildlife (with the exception of birds). Like our dogs and cats, wild animals become accustomed to regular feeding stations. Although well intentioned, this act will cause wild animals to linger in your yard and puts them in danger.
Similarly, don't adopt orphaned or injured animals unless you are a licensed wildlife rehabilitator. Keeping these animals increases your risk of contracting one of the diseases or parasites mentioned above.
Watch for wildlife defecation areas, like communal raccoon latrines. Using proper protective equipment, remove and destroy the feces.
Vaccinations and preventive flea and tick medications are vital in keeping our pets safe from these dangers. Your veterinarian can help you determine your pets' risk factors and then guide you to choosing appropriate vaccines and flea/tick preventives.
The information given here in this article is for general discussion and educational purposes only. You are urged to discuss all aspects of your pet's care with your primary care veterinarian.


text:
Q: I am using a bootstrap modal window to display user alert. I am trying to call the modal window from different places and assigned different content

Alias Name

            <button type="button" class="close" data-dismiss="modal" aria-label="Close">
                ×
            </button>

Do you want your First Name as Alias Name

            <button id="myModalButtonYes" type="button" class="btn btn-primary" data-
   dismiss="modal">Yes</button>
            <button id="myModalButtonNo" type="button" class="btn btn-secondary" data-
   dismiss="modal" >No</button>

redpajama_set_name: RedPajamaStackExchange


No contracts are required and your satisfaction is guaranteed.Give us a call at 1-716-870-0810 for more information or to request a free quote.
Mosquito control is good for business too.
Responsible mosquito pest control solutions that target biting insects help municipalities and businesses keep visitors to outdoor spaces comfortable and safe from insect-borne disease. Mosquito Focus is serious about public safety.
The last thing you want customers to say about your business? "It's that place where the mosquitoes were so bad." For commercial properties with outdoor spaces, eliminating mosquitoes can boost business, ensuring customers' comfort and safety. Golf courses, dining patios, outdoor stages, theme parks – when business relies on customers being comfortable outdoors, Mosquito Focus is the solution.
For municipalities dedicated to providing a safe environment for outdoor activity, mosquito control allows continued enjoyment during warm weather months when demand is highest. Parks, sports fields, fairgrounds, tennis courts, campgrounds – where people expect to enjoy their public spaces, Mosquito Focus can help.
We provide Mosquito and Tick Treatment Services to the following Western New York Communities: Akron, Alden, Allegany, Amherst, Angola, Angola on the Lake, Arkwright, Ashford, Aurora, Barker, Billington Heights, Blasdell, Boston, Brant, Busti, Cambria, Carroll, Carrollton, Cattaraugus, Chautauqua, Cheektowaga, Cherry Creek, City of Buffalo, Clarence, Clarence Center, Clymer, Cold Spring, Colden, Collins, Concord, Conewango, Dayton, Delevan, Depew, Dunkirk, East Aurora, East Otto, Eden, Eggertsville, Ellery, Ellicott, Ellicottville, Ellington, Elma, Elma Center, Evans, Farnham, Farmersville, Franklinville, Fredonia, Freedom, French Creek, Gerry, Gowanda, Grand Island, Grandyle Village, Great Valley, Hamburg, Hanover, Harmony, Harris Hill, Hartland, Hinsdale, Holland, Humphrey, Ischua, Jamestown, Kenmore, Kiantone, Lackawanna, Lake Erie Beach, Lancaster, Leon, Lewiston, Little Valley, Lockport, Lyndon, Machias, Mansfield, Marilla, Middleport, Mina, Napoli, New Albion, Newfane, Newstead, Niagara, Niagara Falls, North Boston, North Collins, North Harmony, North Tonawanda, Olean, Orchard Park, Otto, Pendleton, Perrysburg, Persia, Poland, Pomfret, Portville, Porter, Portland, Randolph, Red House, Ripley, Royalton, Salamanca, Sardinia, Sheridan, Sherman, Sloan, Somerset, South Dayton, South Valley, Springville, Stockton, Tonawanda, Town Line, University at Buffalo, Villenova, Wales, Wanakah, Westfield, West Seneca, Wheatfield, Williamsville, Wilson, Yorkshire, Youngstown and More.


text:
Tenuipalpus tepicanus är en spindeldjursart som beskrevs av De Leon 1957. Tenuipalpus tepicanus ingår i släktet Tenuipalpus och familjen Tenuipalpidae. Inga underarter finns listade i Catalogue of Life.

Källor

Spindeldjur
tepicanus

redpajama_set_name: RedPajamaWikipedia


From Idomeni to Calais, images of people trying to pass borders more and more insuperable get to us. At the same time the European states are reorganising the internal management of immigration through new sorting facilities, and heightening control in administrative detention centres.
On the island of Lesvos, where the deportation machine is working at full speed, all the camps for refugees are cleared out and the registration center 'moria' has showed the true purpose and is now functioning as an deportation centre, NBK is still standing strong and operating at full speed.
In recent weeks, the State and its "good" army have intensified their attacks against our friends in Calais, between two racist pogroms unleashed by fascists. But those who attack libertarian networks should expect strong reactions.
If the houses built in Calais are destroyed, we'll destroy the repressive infrastructure, in Marseille like anywhere else.
Click on image for hi-res version (4.8mb). Poster also available in French.
Read the full communique here, received on March 16th.
For more posters in solidarity with the struggle in Calais click here.
See also: Chronology of resistance from Rabble and posters in solidarity here.
Attack on the French Institute in Athens in solidarity with the persecuted migrants in Calais, against the evacuation of the Jungle!
[Calais] Two posters in solidarity with the struggle – No to all evictions!
At about 10pm on January 10th 2016, a few hours after we occupied the empty building on Themistokleous 58 St. in Exarchia, a guy appeared outside the premises who claimed to represent 'ANASKEVI M EPE' and said that the building belongs to this real estate company.
As long as the racists, motivated by a campaign of fear in the mass media and supported by hypocrite policies, gather and run riot in front of the refugees under the eyes of police, we have the obligation to stand in their path – by all means necessary.
On May 22nd, about 200-250 people took to the streets in Basel in a demonstration against Frontex and the European migration-regime on the occasion of the international action days against the 10 year anniversary of Frontex.


3722 Holland Avenue, Unit 1, Dallas, TX 75219 (MLS # 14067050) | Jami Greenville Real Estate Solutions, L.L.C.
Updated contemporary condo in the heart of Oak Lawn. Tall ceilings greet you at this spacious two bedroom with painted brick and concrete floors. Open floor plan is anchored by a fireplace and features dining space off the kitchen, along with two separate covered outdoor patios. The breakfast bar opens to dining room from a sleek kitchen featuring built-in appliances and ample cabinets for storage space. Expansive master suite features sitting area, two walk-in closets and updated bath with shower; conveniently located washer & dryer. Second bedroom features walk-in closet and full bath with dual sinks, built-in cabinets and tub-shower. Recently installed new roof.
Listing provided by Michelle Wood, Compass RE Texas, LLC.


Have you ever had grilled pizza? Did you even know that was a thing?! Several years ago, I was driving and some DJ was talking about grilling pizza and so I decided to try it. Never having done it before and not having a recipe or anything to follow, we just winged it, but we've been doing it ever since! Something about the grill just brings all the delicious flavors together. If you've never grilled pizza, you are definitely gonna want to try this one!
What you'll need: *Pre-made pizza crust *Olive oil *Pizza sauce (You can make your own, if you're feelin' wild!) *Shredded mozzarella cheese, or pizza blend cheese *Toppings (We usually use turkey pepperoni, mushrooms, green peppers, tomatoes, olives, etc.) Here's what you do: 1. Pre-heat your grill on medium heat for a few minutes 2. Put your pizza crust face down on the grill, and baste the side that is up (which is actually the bottom...have I lost you?!) with olive oil. Let it sit for a few minutes.
3. If you haven't already, this would be a good time to get your toppings ready!
4. Flip the crust over, add your sauce, and spread. Check out those grill marks...mmmm!
5. Sprinkle on your cheese and add other toppings.
6. Cook until the cheese melts and the crust is crisp (5-10 minutes).
7. Go ahead, stuff your face!
We usually choose a thin crust so it is really crisp and delicious.
When we go out for pizza or order in, we always go by the motto that the more crust the better, so I think the thinner crust just makes it feel like a whole different food to us. Bonus: it's even toddler approved!
If you didn't have dinner plans tonight, now you do! Give this one a whirl...you can thank us later!
love it I've never cooked one on the grill will definitely have to give it a try. Thanks for sharing this recipe have a great week.
We've never made grilled pizza before..but that looks awesome!


Principal Resume Template File57100464006 Stunning Sample Principal Resume Samples Related To Resume Templates Qa Resume Sample Assistant Principal Resume Template Uploaded by on Monday, April 15th, 2019 in category executive resume template word.
See also Principal Resume Template Principal Resume Samples Principal Resume Sample Samples Pinterest And Professional Assistant Templates Showcase Your Talent Assistant Principal Resume Template from executive resume template word Topic.
Here we have another image Principal Resume Template A 49 Assistant Principal Resume Resume Principal Resume Template Assistant Principal Resume Template featured under Principal Resume Template File57100464006 Stunning Sample Principal Resume Samples Related To Resume Templates Qa Resume Sample Assistant Principal Resume Template. We hope you enjoyed it and if you want to download the pictures in high quality, simply right click the image and choose "Save As". Thanks for reading Principal Resume Template File57100464006 Stunning Sample Principal Resume Samples Related To Resume Templates Qa Resume Sample Assistant Principal Resume Template.


This 100% natural scented candle 'Be Cool, Be Calm (and keep yourself together)' fills your home with the delicious sweet and fresh scent of flowers and grapefruit. The candle is made with eco soy wax and a mix of ylang ylang, geranium and grapefruit essential oils. The combination of these scents get you right in a peaceful and serene mood. The candle comes in a glass jar with lid.
Eco soy wax burns cleaner and much more efficiently than traditional paraffin wax. It also forms the perfect base for the natural scents we create using pure essential oils. Burning hours: 36 hours.
Made from eco soy wax with ylang ylang, geranium and grapefruit essential oils.


The winning photographs of the second edition of the "Drone Aerial Photography Contest 2015" have announced. These beautiful and inspirational photos taken with a drone were meticulously selected from more than 5000 entries submitted around the world by professionals and amateur drone photo enthusiasts. The competition 2015 was organized by Dronestagram and National Geographic and judged by a panel of experts, say, National Geographic photographer Ken Geiger (Pulitzer Prize-winning photographer), Jean-Pierre Vrignaud (Editor in Chief of National Geographic France) and so on. The rewards will be given away in conjunction with official partners National Geographic, Kodak, Parrot, Go Pro, Hexo+, Picanova, Hobbico and Adobe.


text:
\section{Introduction}
Recent advancements in generative neural networks have greatly improved the quality of image generation \cite{gan, pix2pix, stylegan, cyclegan}. In the domain of video generation, however, the problem becomes much more intricate because the temporal information is introduced as another dimension. Most results produced by the existing methods are still not satisfactory to human eyes \cite{scene_dynamics, convlstm, fgst, deep_video_generation}. Our  proposed approach  focuses on a specific task in the domain of video generation: generating a sequence of frames based on the information provided by a single still image. We name this task ``Single image video prediction''.

\section{Related Work}

\paragraph{Semantic Foreground-Background Distinction.}
For better video generation, some methods have also distinguished the foreground information from the background information. For instance,~\cite{scene_dynamics} explicitly models the background and foreground as two different parts. This is motivated by the observation that the background is relatively static in most videos, and helps the model to learn the motion pattern of objects. This approach of semantic distinction is also useful for video understanding applications, such as in ~\cite{motion_property_understanding}, where   salient movements are detected by taking into account the temporal relationships between pixels of subsequent frames. In another approach ~\cite{scene_transition}, they used the Mask R-CNN ~\cite{mask_rcnn} instead of object proposals to model scene transition by segmenting the region of interest from the background.

\paragraph{Temporal-based Video Prediction.}
Videos are composed of frames  that are temporally related and that follow a sequential order. Hence, most of the previous approaches for video prediction use recurrent neural networks ~\cite{deep_video_generation, mocogan, decompose_motion} or 3D convolutional neural networks~\cite{fgst, videogan, SDCNet} to capture the temporal relationships between the frames. For example, in Villegas et al.~\cite{decompose_motion}, a convolutional LSTM encoder-decoder architecture was used to predict the next frames conditioned on previous frames up to a fixed number of time steps. In ~\cite{SDCNet}, the authors use a spatial-temporal architecture with 3D convolution layers to capture an optical flow map which is used to generate future frames from the first frame.

Amazing results have been achieved in the domain of facial expression generation. In one such approach~\cite{affinegan, ganimation}, an anchor image, as well as a facial expression intensity score as guidance were used to generate corresponding frames for an expression. However, this type of approach requires additional information such as facial landmark cropping ~\cite{affinegan}, or AU annotation ~\cite{ganimation}. Furthermore, note that these kinds of information are very specific, and limited to the domain of facial images. In addition, these methods can only generate videos with pre-defined lengths, depending on how much they want to scatter their intensity score.

 In contrast to those previous works, our model is able to generate frames of an arbitrary length. This provides flexibility in generating the desired intensity of a given target expression, possibly beyond the intensity of expressions exhibited in the training data. Given that our model does not require any guidance in the generation process, this allows us to go beyond the facial expression domain with respect to testing our method on other types of videos.

\section{Method}
\label{sec:method}

\subsection{Two-Phase Training and Testing}
In our setup, given an input image, despite of generating a plausible next frame, the network has two other target tasks. The first task is to generate a semantic mask that marks the main moving object. The second task is to introduce controllablity to the generation process. In order to achieve these goals in an unsupervised manner, we propose a two-phase training strategy, with one target task per phase.

\paragraph{\textbf{Training Phase I.}} In this phase, the network is trained to construct a future frame $f_{t+1}^\prime$ based on the current frame $f_t$, meanwhile targeting at a side task of generating a semantic mask. In our setup, we assume an input frame can be decomposed into a foreground region $f_{fg}$, it marks the main moving object, and a background region $f_{bg}$, where the objects in this region should remain static, and when constructing the successor frame, our intention is to let the network copy the pixels in the background, while to generate new pixels in the foreground. We use a combination of losses \ref{} to ensure the mask generated by the network satisfies the above intention.

\paragraph{\textbf{Training Phase II.}}
In this phase, the network is trained to generate a high-quality future frame from an input frame, conditioned on a mask that indicates the target location of the moving object. We freeze the weights in the mask generator and use it as a pretrained mask extractor, and then finetune the rest of the network. This phase of training will finetune the network to condition it's generation on an input mask. Imagine the location of the object in $f_t$ is at $(x_t, y_t)$, with its corresponding mask $m_t$, similarly, its location in $f_{t+1}$ is at $(x_t + \Delta_x, y_t + \Delta_y)$, with mask $m_{t+1}$, we assume the shape of the mask does not change much between two consecutive frames, so that we can approximate $m_{t+1}$ by shifting the entire $m_t$ by $(\Delta_x, \Delta_y)$:
\begin{align}
    m_{t+1} \approx m_t + (\Delta_x, \Delta_y)
\end{align}
With the above approximation, we can use $m_{t+1}$ for training, this can be obtained by $\mathbb{M}(f_{t+1})$, and $m_t + (\Delta_x, \Delta_y)$ for testing, where $(\Delta_x, \Delta_y)$ can be determined by user's input. Since $\mathbb{M}$ is only able to generate soft masks, it may contain additional information other than the location of the object, we solve this by converting them into binary masks in this phase of training.

\paragraph{\textbf{Test Phase.}}
In this phase, a frame is generated by using two forward passes

\subsection{Model Architecture}
Our model is composed of tow modules: a mask generation network $\mathbb{M}$, and a frame generation network $\mathbb{G}$.

\paragraph{\textbf{Mask Generator.}} The mask generator $\mathbb{M}$ takes a frame $f_t$ as input, and outputs a single-channel binary mask $m$ that marks the foreground and the background of the frame, with 1s indicating foreground pixels and 0s indicating background pixels. Then the mask $m$ is multiplied with $f_t$  to obtain foreground pixels $f_{fg}$, and the inverse mask $(1-m)$ is multiplied with $f_t$ to obtain background pixels $f_{bg}$. We noticed that the foreground marked by the network is usually larger than the moving object, this can be explained that the network is agnostic of the motion that's taken to arrive to the next frame, in other words, when looking at an object in the scene, it could move to all directions with equal possibility, so that the foreground region is of similar shape to the object, but with extended boundaries in all directions.

\paragraph{\textbf{Frame Generator.}} Following the recent success on image generation tasks, we use a VQ-VAE based architecture as our frame generator $\mathbb{G}$, it consists an encoder $E$, a decoder $D$, and a learnable discrete code book $\mathcal{Z} = \{z_k\}^K_{k=1}\subset \mathbb{R}^{n_z}$.

The encoder takes the aforementioned $f_{fg}$ and $f_{bg}$ as input, and encodes them separately into latent codes $z_{fg} = E(f_{fg})$ and $z_{bg} = E(f_{bg})$, after that, we use element-wise quantizatoin $\mathsf{q}(\cdot)$ to map them onto the closest entry in the code book:
\begin{align}
    z_{\mathsf{q}} = \mathsf{q}(z) := \left( \underset{z_k \in \mathcal{Z}}{\arg\min } \lVert z-z_k \rVert \right) \in \mathbb{R}^{h \times w \times n_z}
\end{align}
The construction of the next frame  by $\hat{f_{t+1}} \approx f_{t+1}$ is given by:
\begin{align}
    \hat{f_{t+1}} = D(\mathsf{q}(z_{fg} + z_{bg}))
\end{align}
During training, the generator is also equipped with a discriminator $\mathbb{D}$ to boost the generation quality \cite{taming}. The discriminator is only used in the second phase of training \ref{}, and it only starts after a certain trianing steps.

\subsection{Training Objectives}
During the training phase, given an input frame, we are aiming at three distinctive goals:
\begin{itemize}
    \item Generating a meaningful mask that separates the foreground and the background of the frame.
    \item Generating a plausible next frame.
    \item The generated next frame should be conditioned on the generated mask.
\end{itemize}
Our model is trained in an unsupervised, end-to-end fashion. To achieve the above mentioned goals, we use a combination of different objectives.

\paragraph{\textbf{Mask Losses.}} To generate semantic meaningful masks, first we use a background loss:
\begin{align}
    \mathcal{L}_{bg} = \lVert (1-m) \odot f_t - (1-m) \odot f_{t+1}\rVert_1
\end{align}
where mask is generated by the network using the next frame $f_{t+1}$, since the network by default generates the mask that mark the foreground, here we use $1-mask$ as the inversed mask to mark the background. This loss enforces that the generated mask should mask out the correct background since we assume the content in the background doesn't change much between two consecutive frames, and this is the main objective that constrains the network to generate meaningful masks.

However, if we solely rely on this loss, there's a potential pitfall where the network will generate a masks that mark everything to be the foreground, such that the inverse mask will be all zero, in this case, this objective is easily optimized. To avoid this, we use a loss to shrink the mask as much as possible, to do that, we first compute a reference mask size $\Bar{d}$, where $d$ is given by:

\begin{align}
    d=
\begin{cases}
    1,  & \text{if } (f_{t+1} - f_t) > \beta\\
    0,  & \text{otherwise}
\end{cases}
\end{align}
where $\beta$ is a changeable threshold. Then, we use this reference mask size to compute the mask sparsity loss:
\begin{align}
    \mathcal{L}_{spar} = \max \{\lVert \Bar{m} - \Bar{d}) \rVert_1, 0\}
\end{align}
The sparsity loss will counter the background loss to shrink the generated mask as much as possible. The weight ratio between $\mathcal{L}_{sparsity}$ and $\mathcal{L}_{bg}$ is critical for generating high quality masks, we will further discuss this in the experiment section.

In addition, since our network is unable to generate hard masks, to avoid soft and imprecise masks, we employ another loss to encourage binerization of the mask.
\begin{align}
    \mathcal{L}_{binary} = \min\{m, (1-m)\}
\end{align}

Our final objective for generating semantic masks is a weighted sum of all three losses mentioned above:
\begin{align}
    \mathcal{L}_{mask} = \lambda_{1} \mathcal{L}_{BG}+ \lambda_{2} \mathcal{L}_{sparsity} + \lambda_{3} \mathcal{L}_{binary}
\end{align}
This mask objective is only used in training phase I, and we dynamically update $\lambda_2$ during training, such that the ratio between the background loss and the sparsity loss fits to a linear model with a negative slop.

\begin{align}
    \mathcal{L}_{GAN}(\mathbb{G}, \mathbb{D}_g, \mathbb{D}_l) =  & \mathbb{E}_{f_{t+1}}[\text{log}\mathbb{D}_g(f_t, f_{t+1})] +
    \nonumber\\ &\mathbb{E}_{f_{t+1}}[\text{log}\mathbb{D}_l(f_t, f_{t+1})] +
    \nonumber\\ &\mathbb{E}_{f_{t}}[\text{log}(1-\mathbb{D}_g(f_t, \mathbb{G}(f_t))] +
    \nonumber\\ &\mathbb{E}_{f_{t}}[\text{log}(1-\mathbb{D}_l(f_t, \mathbb{G}(f_t))]
\end{align}
where $\mathbb{G}$ tries to minimize this objective against its adversaries $\mathbb{D}_g$ and $\mathbb{D}_l$ that try to maximize it:
\begin{align}
    \mathbb{G}^\ast =  \text{arg }\! \min_\mathbb{G}  \max_{\mathbb{D}_g, \mathbb{D}_l}
    \mathcal{L}_{GAN}(\mathbb{G}, \mathbb{D}_g, \mathbb{D}_l)
\end{align}

\subsection{Implementation}
As discussed above, we have two streams in our generator $\mathbb{G}$: a function $m(\cdot)$ that outputs a mask, and a function $d(\cdot)$ that generates the difference map. For $m(\cdot)$, we use a ResNet \cite{ResNet} based architecture with two downsampling/upsampling layers scattered between 9 residual blocks, and we use the $tanh$ function to rescale the output to  $[0,1]$. For $d(\cdot)$, we employ an 8-layer U-Net \cite{unet} with skip connections to enable the flow of both low-level and high-level features. We use the PatchGAN \cite{pix2pix} discriminator with $70\times70$ patch size for both the global and local discriminators. We set the video batch size to 1 and sample 10 frames from each individual clip for training. We use the Adam \cite{Adam} optimizer and set the learning rate to 0.0002, the learning rate linearly decays to 0 starting from the half of the total epochs. The scheduled sampling process starts at the half of the total epochs in the training. We set our initial sampling rate to 0.1 and increase this by 0.1 for every 20 epochs. All input frames are resized to $256\times256$ and the values normalized to $[-1, 1]$. We will make the code publicly available upon acceptance of the paper.

\begin{figure*}
\phantomsection
\begin{center}
\includegraphics[width=0.93\linewidth]{comparesion.pdf}
\end{center}
   \caption{Qualitative comparison with different models for the ``Happiness'' class on the test set.  The first row are the ground truth frames, the following rows are frames generated using different models. We also compare our results with the AffineGAN, which is not a temporal-based model. Our results are clearly better than ConvLSTM and FGST, and are comparable to results produced by AffineGAN. }
\label{comparesion}
\end{figure*}

\section{Experiments}
\label{sec:sec4}

In this section, we provide detailed analysis and results of our model on different perspectives, including an ablation study on the effects of different components of our model design, and comparison to other methods in the literature.
\subsection{Datasets}
\paragraph{BAIR Robot Pushing Dataset.}

\subsection{Quantitative Results}

\subsection{Qualitative Results}

\section{Conclusion}

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
\label{sec:intro}

Advances in deep generative models have led to impressive results in image and video synthesis.
Typical forms of such models, including Variational Autoencoder (VAE)~\cite{KingmaVAE}, Generative Adversarial (GAN)~\cite{GoodfellowGAN} and recurrent (RNN)~\cite{OordPixelRNN} formulations, can produce complex and highly realistic content.
However, synthesis of realistic images/videos, without the ability to control the depicted content in them, has limited practical utility.
This has led to a variety of conditional generative tasks and formulations.

In the image domain, both coarse- (\eg, sentence \cite{Zhang_ICCV17}) and fine-level (\eg, layout \cite{Zhao_CVPR19} and instance attribute \cite{Frolov_GCP21}) control signals have been explored.
The progress on the video side, on the other hand, has generally been more modest, in part due to an added challenge of synthesizing temporally coherent content.
{\em Future frame prediction}~\cite{Denton_ICML2018,finn_NIPS16,franceschi_ICML20,Kumar_ICLR20,mathieu_ICLR16,vondrick_ICLR17,vondrick_NIPS17,vondrick_CVPR17,Weissenborn_ICLR20} conditions the future generated frames on one (or a couple) seed frame(s).
But this provides very limited control as the object(s), or person(s), depicted in the conditioned frame can move in a multitude of ways, particularly as predictions are made longer into the future.
To address this, a number of methods condition future frames on the action \cite{He_ECCV18,Tulyakov_CVPR18} and object \cite{Nawhal_ECCV20} label.
Still, they only provide very coarse global video-level control.

\begin{figure}
    \centering
    \vspace{2.5em}
    \includegraphics[width=0.85\linewidth]{fig/front_fig.pdf}
    \caption{{\bf Layered Controllable Video Generation.}
    Illustration of the proposed task, where a frame at time $t$ is first decomposed into a foreground/background layers, using the learned mask network $\mathcal{M}$, and then the user is allowed to modify this mask with control signals $\theta$ (\eg, by shifting it by $\Delta_t$) to control the next generated frame realized by generator $\mathcal{G}$. The foreground source and target mask are illustrated in red and blue.}
    \label{fig:front_fig}
\end{figure}

More recent approaches focus on the ability to control the video content on-the-fly at the frame-level.
Typically, these methods are formulated as conditional auto-regressive (or recurrent) models that generate one frame at a time, conditioned, for example, on the discrete action label~\cite{Kim_CVPR20} or keypoint-based human pose specification~\cite{Villegas_ICML17,Walker_ICCV17,Yang_ECCV18} (\eg, obtained from a target video source \cite{Zablotskaia_BMVC19}).
Such methods, however, require dense per-frame annotation of actions or poses at training time, which are costly to obtain and make it challenging to employ such approaches in realistic environments.
The task of {\em playable video generation}~\cite{playable}, has been introduced to address these limitations.
In playable video generation, the discrete action space is {\em discovered} in an unsupervised manner and can then be used as a conditioning signal in an auto-regressive probabilistic generative model.
While obtaining impressive results, with no intermediate supervision and allowing frame-level control over the generative process,~\cite{playable} is inherently limited to a single subject and a small set of discrete action controls.

Thus, in this work, we aim to allow richer and more granular control over the generated video content, while similarly requiring no supervision of any kind -- \ie, having only raw videos as training input, as in~\cite{playable}.
To do so, we make an observation (inspired by early works in vision \cite{adelson_PAMI94,jojic_CVPR01,kumar_IJCV08}) that a video can be effectively decomposed into foreground / background layers.
The background layer corresponds to static (or slowly changing) parts of the scene.
The foreground layer, on the other hand, evolves as the dominant objects move. %
Importantly, the foreground mask itself contains position, size and shape information necessary to both characterize and render the moving objects appropriately.
Hence, this foreground mask can in itself be leveraged as a useful level of abstraction that allows both intuitive and simple control over the generative process.

With the above intuition, we therefore propose an approach that automatically learns to segment video frames into foreground and background, and at the same time
generates future frames conditioned on the segmentation and the past frame, iteratively.
To allow on-the-fly control of video generation, we expose the foreground mask to the user for manipulation -- \eg, translations of the mask leads to corresponding in-plane motion of the object, resizing it would lead to the depth away/towards motion, and changes in the shape of the mask can control the object pose.

From the technical perspective, the challenge of formulating such a method is two fold.
First, unsupervised foreground/background layer segmentation is highly ambiguous, particularly if the background is not assumed to be static and the foreground motions can be small.
Second, user input needs to be anticipated to ensure model learns how to {\em react} to changes in the mask, without explicit access to such information.
To this end, we propose a two-stage learning procedure.
In the first stage, the network learns how to perform foreground/background separation and, conditioned on this layered representation, future frame prediction.
Specifically, we introduce a set of sophisticated losses and a dynamic prior to learn how to predict a foreground mask and leverage VQ-VAE~\cite{VQVAE} to predict foreground and background latent content which is then fused and decoded to the next frame.
In the second stage, we simulate user input and fine-tune the generative model such that this user input can be appropriately handled.

\vspace{0.1in}
\noindent
{\bf Contributions:} Our contributions are multi-fold.
\vspace{-.5em}
\begin{itemize}[leftmargin=*]
    \item
    From raw video data, our model learns to generate foreground/background separation masks in an unsupervised manner.
    We then leverage the foreground layer as a flexible (parametric) user control mechanism for the generative process.
    This provides both richer and more intuitive control compared to action vectors \cite{playable} or sparse trajectories \cite{HaoCVPR18}.

    \item To effectively train our model we introduce two-stage training: the first stage tasked with learning how to separate layers and perform future frame prediction; the second, to adopting and anticipating user control.
    \item To prevent over-/under-segmentation, we regularize layer separation with sparsity loss and dynamic mask size prior.
    \item Finally, we validate our approach on multiple datasets and show that we are able to generate state-of-the-art results and, at the same time, allow higher level of control over the generated content without any supervision.
\vspace{-.25em}
\end{itemize}

\section{Related works}
\label{sec:related}
\vspace{-0.5em}
\paragraph{Video generation}
Early video generation techniques proposed to generate a video as a whole. Most of these convert a noise vector, sampled from simple distribution or a prior, to a video, using a GAN \cite{vondrick_NIPS17,Acharya_2018,Wang_CVPR2020} or VAE \cite{Babaeizadeh_2018,Denton_ICML2018} formulation. More recent architectures leveraged transformer-based formulations \cite{Yan_2021,Neimark_2021,Rakhimov_2021,Weissenborn_ICLR20} that have generally resulted in higher quality video outputs. As an alternative to 3D (transposed) convolution techniques, that generate all frames at once, recurrent auto-regressive variants \cite{Kalchbrenner_2017} have also been explored.
While these early works largely focused on the video quality and resolution, more recently, the focus has shifted to conditioned or controlled video generation.

\input{fig/pipeline_item}

\paragraph{Video generation with global control}
Future frame prediction considers the task of generating a video conditioned on a few starting seed frames. Early approaches to future prediction employed deterministic models \cite{finn_NIPS16,mathieu_ICLR16,vondrick_ICLR17,Wichers2018HierarchicalLV} that failed to model uncertainty in the future induced by variability of unfolding events. To overcome this limitation, later methods, based on VAE \cite{Babaeizadeh_2018,Denton_ICML2018}, GAN \cite{Lee_2018,Kwon_CVPR2019}, and probabilistic formulations \cite{Xue2019VisualDS}, attempted to introduce real-world stochasticity into the generative process. Action label conditioning, in combination with seed frames or not, where a video sequence is generated conditioned on an input action label \cite{Kim_Neurips2019,Wang_WACV2020} is also popular; some such approaches leverage disentangled factored representations (\eg, of subject identity and action \cite{He_ECCV18}). Other types of global conditioning signals include action-object tuples \cite{Nawhal_ECCV20}. However, these methods, collectively, require action annotations for training, and, more importantly, do not allow control at the frame-level.

\paragraph{Video generation with frame-level control}
More granular control, at the frame-level, has also been explored. Pose-guided generative models first generate a sequence sparse \cite{Walker_ICCV17,Villegas_ICML17,Yang_ECCV18} or dense \cite{Zablotskaia_BMVC19} keypoint human poses, either predictively \cite{Villegas_ICML17,Walker_ICCV17} or from a source video \cite{Zablotskaia_BMVC19}, and then use these for conditional generation of respective video frames. However, these methods are only applicable to videos of human subjects and require either pose annotations or a pre-trained pose detector. Alternatively, individual frames can be conditioned on action labels \cite{Kim_CVPR20,Chiappa_2017,Oh_NIPS2015}. However, these methods require dense frame-level annotations, which are only available in limited environments such as video games. Closest to our work, Menapace \etal proposed Playable Video Generation \cite{playable}, which, in an unsupervised manner, discovers semantically consistent actions meanwhile generating the next frame conditioned on the past frames and an input action, thus providing user control to the generation process. However, their method is limited to a small set of discrete action controls and explicitly assumes a single moving object. In contrast, our method allows richer and more granular control, and can be used to generate, and control, videos with multiple objects.

\paragraph{Video generation with pixel-level control}
It is worth mentioning that some prior works attempted to use dense semantic segmentation \cite{Pan_CVPR2019,WangNIPS19} and sparse trajectories \cite{HaoCVPR18} to control video generation. While such approaches allow granular control at a pixel-level, these representations are incredibly difficult for a user to produce or modify. We also use a form of (foreground) segmentation for control, however, it is unsupervised, class-agnostic, and can be easily controlled either parametrically or non-parametrically.

\paragraph{Unsupervised object segmentation}
Layered representations have long history in computer vision \cite{adelson_PAMI94,jojic_CVPR01,kumar_IJCV08}, and are supported by neuroscience evidence \cite{WebsterBrain}.
Traditional techniques for this rely on feature clustering \cite{Achanta_2012,Alexe_2010,Hochbaum_2009} and statistical background modeling and subtraction \cite{BouwmansBGBook,ElgammalECCV00,StaufferCVPR99}.
Such techniques work best for videos where the background is (mostly) static, lighting fixed and the foreground is fast moving; we refer readers to \cite{Changedetection} for an extensive analysis and discussion.
More recent techniques have focused on generative formulations for the task.
In particular, Bielski \etal proposed the PerturbGAN \cite{Bielski_NeurIPS2019}, their model generates the foreground and background layers separately, and uses a perturbation strategy to enforce the generation of semantically meaningful masks.
Related, MarioNette \cite{SmirnovNIPS21} learns to decompose scenes into a background and a learned dictionary of sprites. Other approaches focus on separation of videos into natural layers (\eg, to factorize secondary effects such as shadows and reflections \cite{alayrac2019visual}) and to control which layer to attend to \cite{AlayracICCV2019}.
Similar to~\cite{vondrick_NIPS17}, we decompose frames and separately model foreground/background content that is then composed/fused together to produce video.
However, unlike~\cite{vondrick_NIPS17} and others, we allow the user to have frame-level control over the foreground mask using both intuitive parametric and nonparametric controls.

\section{Conclusions}
\vspace{-0.02in}

We have introduced layered controllable video generation, an unsupervised method that decomposes frames into foreground and background, with which the user can control the generative process at a frame-level by altering the foreground mask.
Our core contributions are the framework itself, and the two-stage training strategy that allows our model to learn to both separate and control on its own.
We show that various degrees of control can be implemented with our method, from parametric (position, affine) to complete non-parametric
control with the mask.
Our results on \textit{BAIR} and \textit{Tennis} datasets show that our method outperforms the state-of-the-art in both quality and control.

\section{Implementation Details}
\label{sec:A}
\paragraph{Architecture details}
Our model consists of four main modules: the \textit{Mask Network} $\mathcal{M}$, the \textit{Encoder} $\mathcal{E}$, the learnable discrete code book $\mathcal{Z}$, and the \textit{Decoder} $\mathcal{D}$.
We summarize the details in \Table{architecture}.
In total, our model has $47.8$ M parameters, and takes about $191.2$ MB to store on disk.
\input{tab/architecture}

\paragraph{Training Details}
In all our experiments, we use the Adam Optimizer with a fixed learning rate of $4.5\times 10^{-6}$, which we found empirically.
\begin{itemize}[leftmargin=*, noitemsep]
    \item \textit{Initial training for mask-based generation:}
    For this stage, we train all models for $10,000$ iterations.
    For experiments on the \textit{BAIR} dataset, we initialize the ratio between $\lambda_{\text{bg}} : \lambda_{\text{fg}}$ to be $80 : 1$, and reduce it by a factor of $2$ for every $1,000$ iterations until this ratio becomes $5:1$.
    For experiments on the \textit{Tennis} dataset, this ratio is initialized to  $10:1$ and reduced by half after $5,000$ iterations.

    \item \textit{Fine-tuning for controllability:} In this stage of training, we drop all losses related to the mask regularization as their gradient becomes zero (see \Section{phaseII}), except for $\mathcal{L}_\text{bg}$.
    On the \textit{BAIR} dataset, $\lambda_{\text{bg}}$ is set to $5$, and on the \textit{Tennis} dataset, it's set to $0.5$.

    \item \textit{Finding the ``ground-truth'' control signal:} In Equation.~\ref{eq:control}, we use the fully differentiable transformation operation $\mathcal{T}(\theta^t)$ to approximate the mask of the  ground truth next frame using the mask of the current frame.
    For each frame, we optimize the affine transformation matrix that's been used to transform $\mathbf{m}^t$ into $\mathbf{m}_c^t$, the optimization is performed by the ADAM optimizer for $1,000$ iterations with the learning rate of $0.1$.

On the \textit{BAIR} dataset, our model requires $80$GB GPU memory and $40$ hours to train on $4\times$ RTX-6000 (24GB V-RAM per GPU) GPUs, and on the \textit{Tennis} daaset, the model takes $22$GB GPU memory and $23$ hours to train on a single RTX-3090 (24GB V-RAM).
\end{itemize}

\section{Additional qualitative results}
\label{sec:B}

\paragraph{Comparing with other baselines}
In \Figure{bair_compare} and \Figure{tennis_compare}, we show the comparison of the three variants of our model against other baselines that we mentioned in the main text, on the \textit{BAIR} and \textit{Tennis} datasets respectively.
Noticeably on both datsets, our model achieves the most precise control over the generated video (see how the generated motion corresponds to the ground truth sequences), meanwhile generating one of the best frame-quality videos.
For example, CADDY fails to control the robot arm, SAVP+ controls it up to some degree, but as shown in frames $t{=}10$ and $t{=}15$, their control is not as precise, whereas all of our variants provide highly accurate control.

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/bair_compare.pdf}
    \caption{
        Comparing with other baselines on the \textit{BAIR} dataset.
    }
    \label{fig:bair_compare}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/tennis_compare.pdf}
    \caption{
        Comparing with other baselines on the \textit{Tennis} dataset.
    }
    \label{fig:tennis_compare}
\end{figure*}

\paragraph{Generating videos conditioned on user inputs}
In \Figure{control_test_bair} and \Figure{control_test_tennis}, we show how the video generated by our model responds to the user inputs, on the \textit{BAIR} and \textit{Tennis} datasets respectively. Here, for each sample, we keep applying the same control signal (moving left/right/up/down) to the model so that it can generate video sequences with single consistent motions.

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/control_test_bair.pdf}
    \caption{
        Generating videos conditioned on user inputs, \textit{BAIR} dataset.
    }
    \label{fig:control_test_bair}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/control_test_tennis.pdf}
    \caption{
        Generating videos conditioned on user inputs, \textit{Tennis} dataset.
    }
    \label{fig:control_test_tennis}
\end{figure*}

\begin{figure*}[t!]
    \centering
    \includegraphics[width=1\linewidth]{fig/mimic.pdf}
    \caption{
        ``Mimicking''. Our model can be used to extract the motion from a driving sequence and then apply onto different appearances (staring frames). The mask sequence is extracted from the driving video (first row), then applied onto two other staring frames (third, fourth row).
    }
    \label{fig:mimic}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/onestage.pdf}
    \caption{
        Single stage training results in poor testing performance when control is introduced. {\color{dark_green}Green} represents GT next frame mask, {\color{orange_bright}orange} represents generated current frame's mask, and {\color{purple}purple} represents transformed mask. On the top row, the model is tested using ground truth masks to generate future frames (same as how the model was trained). On the bottom row, the model is tested using pseudo control signals (same as real-world testing). The performance of the single stage trained model drops significantly when there are edits to the mask.
    }
    \label{fig:onestage}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{fig/softmask.pdf}
    \caption{Soft masks result in degenerated frames.}
    \label{fig:softmask}
\end{figure*}

\paragraph{Action mimicking}
In \Figure{mimic}, we show that our method can be used to extract the motion from a driving sequence and then apply onto different appearances (staring frames).

\makeatletter
\setlength{\@fptop}{0pt}
\makeatother

\paragraph{Video results}
For more video results, please refer to \href{https://gabriel-huang.github.io/layered_controllable_video_generation/}{this website}.
, where we demonstrate all above mentioned results in video format, as well as other applications of our method, such as generating videos of multi players, real-time user controllable generation demo, and animating a single frame with different motions.

\section{Additional Ablation results - Single Stage Training}
\label{sec:C}
As mentioned in our main text, breaking down our training procedure into two stages is a crucial design in our setup. In stage I, our model focuses on generating a FG/BG segmentation mask, and in stage II, we make edits to this mask to finetune the network for controlled generation.
Training the model in one single stage wouldn't allow us to introduce controllability, without stage II finetuning, the model is extremely vulnerable to any changes to the mask. As shown on Figure \ref{fig:onestage}, on the top row, the model is tested using ground truth masks to generate future frames (same as single stage training setup), and the model performed well. However, on the bottom row, when we try to use pseudo control signals to transform the mask, then use the transformed mask to condition the generation (same as real-world testing for controlled video generation), the model performs poorly.

\section{Additional Ablation results - Soft Masks}
\label{sec:D}
As shown in Fig.~\ref{fig:softmask}, without mask binarization, information is leaked through the soft masks, shifting the mask will not only affect the foreground, but the entire scene, which results in degeneration of the frame.

\section{Experiments}
\label{sec:sec4}

While our method is {\em not} limited to a ``single-agent'' assumption, \ie, single dominant moving agent in the scene, previous work, and notably \cite{playable} which is the closest and the most competitive baseline, are.
Hence, for fair comparison, we adopt the single-agent setup for the majority of our experiments.
We train / test on the following datasets:

\paragraph{BAIR Robot Pushing Dataset \cite{bair}}
This dataset contains 44K video clips ($256 \times 256$ resolution) of a single robot arm agent pushing toys on a flat surface.

\paragraph{Tennis Dataset \cite{playable}} This dataset contains 900 clips extracted from two full tennis matches on YouTube.
These clips are cropped such that only the half of the court is visible.
The resolution of each frame is ($96 \times 256$).

\subsection{Results}
\label{sec:results}

\vspace{-0.5em}
\paragraph{Evaluation Protocol}
We compare our model against other conditional generative methods, focusing on quality of reconstructed sequences and controlability.
We evaluate our model under three control protocols: two parametric (position, affine) and one non-parametric (direct non-differentiable control over mask):
\vspace{-0.06in}
\begin{itemize}[leftmargin=*, noitemsep]
    \item Ours /w \texttt{position} control: We first use our trained mask network $\mathcal{M}$ to extract masks from the ground truth test sequences, then we use Eq.~(\ref{eq:control}) to approximate those masks with control for generation, where $\theta$ is restricted to \texttt{positional} parameters, \ie, $x$ and $y$ translation.
    \item Ours /w \texttt{affine} control: Similar to above, here our $\theta$ employs full \texttt{affine} transformation parameters, \ie, translation, rotation, scaling and shearing.
    \item Ours /w \texttt{non-param} control: We use masks predicted from ground truth test sequences themselves to condition our generation.
    These masks can change at a pixel-level, hence constituting non-parametric control.
\end{itemize}
\vspace{-0.06in}
For testing, we generate video sequences conditioned on the first frame $f_0$ and the user input $\Theta = \{\theta^t_u\}_{t=1}^T$ in all cases.

\paragraph{Metrics} To quantitatively evaluate our results, we consider standard metrics:
\vspace{-0.06in}
\begin{itemize}[leftmargin=*, noitemsep]
  \item \textit{Learned Perceptual Image Patch Similarity (LPIPS)}~\cite{LPIPS}: LPIPS measures the perceptual distance between generated and ground truth frames.
  \item \textit{Fréchet Inception Distance (FID)}~\cite{FID}: FID calculates the Fréchet distance between multivariate Gaussians fitted to the feature space of the Inception-v3 network of generated and ground truth frames.
  \item \textit{Fréchet Video Distance (FVD)}~\cite{FVD}: FVD extends FID to the video domain. In addition to the quality of each frame, FVD also evaluates the temporal coherence between generated and ground truth sequences.
  \item \textit{Average Detection Distance (ADD)}~\cite{playable}: ADD first uses Faster-RCNN~\cite{fasterrcnn} to detect the target object in both generated and ground truth frames, then calculates the Euclidean distance between the bound box centers.
  \item \textit{Missing Detection Rate (MDR)}~\cite{playable}: MDR reports percentage of unsuccessful detections in generated vs. successful detections in ground truth sequences.
  \item \textit{Rooted Mean Square Error of Displacement (RMSED)}~\cite{playable}: RMSED, which we define, reports the RMSE of the displacement of ground truth locations vs. generated locations. See Figure~\ref{fig:compare_bair} for more details.
\end{itemize}
LIPIPS, FID and FVD measure the quality of generated videos. ADD and MDR measure how the action label conditions the generated video, and RMSED measures the precision of control.

\paragraph{Baselines}
CADDY \cite{playable} is the only unsupervised video generation method that allows frame-level user conditioning, thus we use it as our main baseline.
We also include results of other frame-level conditioned methods: MoCoGAN \cite{Tulyakov_CVPR18}, SAVP \cite{Lee_2018}, and their high-resolution adaptations MoCoGAN+ and SAVP+ from \cite{playable}.

\input{tab/bair.tex}
\input{tab/tennis.tex}

\paragraph{Quantitative Results} We report the results on the \textit{BAIR} dataset in Table~\ref{tab:bair}.
We highlight that in terms of RMSED score, our method achieved the highest precision of control (more than $\times5$ improvement compared to other baselines). In terms of generation quality,  with similar level of abstraction of ground truth information (ours: 6 continuous \texttt{affine} control parameters, CADDY: 7 discrete action labels), our model outperformed CADDY on all three evaluated metrics by a large margin, demonstrating that our model is of better generation quality. With \texttt{non-param} control, our generation quality is comparable to the SAVP+.

Table~\ref{tab:tennis} shows the results on the \textit{Tennis} dataset. In terms of generation quality (LPIPS, FID, FVD), all our adaptations outperformed all other comparing methods.
Specifically, in terms of FID score, our model is up to 37\% better than the closest baseline (\cite{playable}).
Further, in terms of control precision, our method achieves the lowest error on ADD and MDR (improvement of 80\% \& 70\% respectively), indicating our method is able to generate consistent players with accurate control.
One can see that simple positional control works much better here compared to the \textit{BAIR} dataset. This can be attributed to largely in plane motion of the subject.

\input{fig/compare_bair}
\input{fig/compare_tennis}
\input{fig/control_item}

\paragraph{Qualitative Results}
In Figure~\ref{fig:compare_bair} and Figure~\ref{fig:compare_tennis}, we show generated sequences on the \textit{BAIR} and \textit{Tennis} dataset (we used our model with \texttt{affine} control for both cases shown). In terms of image quality, our method is superior to competitors. In terms of control accuracy, unlike other competing methods, our method is able to precisely place the robot arm and the tennis player in the correct position.

In Figure~\ref{fig:control}, we show the results of our model reacting to different user control signals. On the \textit{Tennis} dataset, our method not only moves the player in the correct direction, it's also able to generate plausible motions of the player itself. On the \textit{BAIR} dataset, our model is able to ``hullcinate'' what's missing in the original frame and generate frames with respect to the control signal, \ie, in the ``down, 35 pixels'' example, our model successfully generates the upper part of the robot arm, not available on the input frame.

\input{fig/fancy_item}

In Figure~\ref{fig:fancy}, we show that our method is capable of generating and controlling videos with multiple moving objects by simply overlaying two individually controlled mask sequences together, \ie, producing 2 and 3 players in this example. As far as we know, our method is the only video generation method that allows frame-level control of multiple objects acting in the same scene. We provide more visual results in the \textit{Supp. Mat.}\xspace

\subsection{Ablation Study}
\label{sec:ablation}

\input{fig/ablation}
\textbf{Mask losses.}
Here we explore impact of our
key design choices have on the quality of generated results and the foreground mask.
We show quantitative and qualitative results in Figure ~\ref{fig:ablation_mask}.
The background loss $\loss{\text{bg}}$ enforces the network to generate meaningful masks, without it, the mask network fails to generate a reasonable mask (all zeros).
The foreground constraint $\loss{\text{fg}}$ shrinks the mask as much as possible. Without this term the network learns a travail solution, where $\loss{\text{bg}}$ in Eq.(\ref{eq:loss_bg}) becomes 0 -- labeling everything as the foreground (all ones).
When computing the foreground loss $\loss{\text{fg}}$, we introduce a dynamic mask size prior. We ablate this choice by instead using a fixed global prior of $0.15$ as in \cite{Bielski_NeurIPS2019}.
Visuals show that if we do not use dynamic prior, the network tends to generate masks with a fixed size, which leads to hollow masks for samples with larger foreground.
To prevent information leaking from soft masks, we binarize the masks with thresholding the mask value, without the binary loss $\loss{\text{bin}}$, some pixels on the mask fails to pass the threshold and leave some defects on the binarized mask.
Overall, the ablations show that all our design choices are important. %

\paragraph{One-stage training VS. Two-stage training}
\label{sec:ablation}
Breaking our training procedure into two stages is a crucial design for the performance of our method. As described in \Eq{optimization}, a well-trained mask generator is a prerequisite for finding the \textit{pseudo user control $\hat\boldsymbol{\theta}$}, which we use to introduce controllability to our model. Nevertheless, we still experimented with training the model with one single shot (training Stage II directly by replacing $\mathbf{f}^t$ with $\mathbf{f}^{t+1}$).
This leads to vastly poorer performance during test time (Figure \ref{fig:ablation_mask}, ``single-stage training").

\section{Method}
\label{sec:method}

We first discuss our framework in detail then our two-stage training strategy.

\subsection{Framework}

\vspace{-0.5em}
\paragraph{Inference}
For easy explanation, we first explain the inference flow of our method.
As illustrated in \Figure{pipeline}~(b), our method is composed of mainly two components: the mask network and the generator network.
The \textit{mask network} $\mathcal{M}$ takes as input an image frame $\mathbf{f}^t\in\mathbb{R}^{3 \times W\times H}$ at time $t$ and outputs a mask:
\begin{equation}
    \mathbf{m}^t=\mathcal{M}\left(\mathbf{f}^t\right)
    ,
    \label{eq:masknet}
\end{equation}
where $\mathbf{m}^t \in \mathbb{R}^{W\times H}$,
segmenting the foreground layer and the background layer.
Then, our \textit{generator network} $\mathcal{G}$ takes the mask and the frame as input to generate the next frame:
\begin{equation}
    \hat{\mathbf{f}}^{t+1}=\mathcal{G}\left(\mathbf{f}^t, \mathbf{m}^t\right)
    .
    \label{eq:gennet}
\end{equation}
More specifically, for the generator, we employ a VQ-VAE framework \cite{VQVAE} for its high-quality image generation.
Hence, as shown in \Figure{G}, the generator $\mathcal{G}$ can be written as a composite function of an encoder $\mathcal{E}$, a decoder $\mathcal{D}$, and a learnable discrete code book $\mathcal{Z} = \{\mathbf{z}_k\}^K_{k=1}\subset \mathbb{R}^{n_z}$, where $n_z$ is the dimension of each code, we encode the foreground layer and the background layer separately in our pipeline, and then merge them in the latent space before feeding them into the decoder:
\begin{equation}
\begin{split}
    \mathbf{z}_{fg}^t &=\mathcal{E}\left(\mathbf{f}^t \odot \mathbf{m}^t\right) \\
    \mathbf{z}_{bg}^t &=\mathcal{E}\left(\mathbf{f}^t \odot (1-\mathbf{m}^t)\right) \\
    \hat{\mathbf{f}}^{t+1} &=\mathcal{D}\left(
        \mathsf{q}\left(\mathbf{z}_{fg}^t\right)
        +
        \mathsf{q}\left(\mathbf{z}_{bg}^t\right)
    \right)
\end{split}
    \;,
    \label{eq:vqvae}
\end{equation}
where $\odot$ is the element-wise product and $\mathsf{q}$ is the element-wise quantization function defined as:
\begin{equation}
    \mathsf{q}(\mathbf{z})_{ij} :=
        \underset{\mathbf{z}_k \in \mathcal{Z}}{\arg\min }
        \left\| \mathbf{z}_{ij}-\mathbf{z}_k \right\|
    ,
\end{equation}
where $i$ and $j$ are the row and column indices.
$\mathcal{G}$ and $\mathcal{M}$ are then used to generate the video via auto-regression.

\input{fig/G_item}

\paragraph{Control}
With the above inference pipeline, we enable on-the-fly user control by modifying the mask to create $\mathbf{m}^t_c$ and using it in place of $\mathbf{m}^t$ in \Eq{gennet}.
Mathematically, we write
\begin{equation}
    \mathbf{m}^t_c = \mathcal{T}\left(\mathbf{m}^t, \boldsymbol{\theta}^t\right)
    ,
    \label{eq:control}
\end{equation}
where $\mathcal{T}$ is the mask controlling operation parameterized by $\boldsymbol{\theta}^t$.
In the crudest form, the controlling operation $\mathcal{T}$ can simply be shifting the mask $\mathbf{m}^t$ by $\Delta x$ and $\Delta y$  in horizontal and vertical directions, respectively, or for example, be an affine transformation.
Both can be implemented differentiably as a simple parametric coordinate transformation on $\mathbf{m}^t$~\cite{Jadeberg15,Jiang19}, allowing differentiable control of the mask.
We utilize this the differentiabilty later in \Section{results} to find the ``ground-truth'' control signal for easy comparison with existing works.
Other forms of control are also possible, including more granular non-parametric, and non-differentiable, direct manipulation of the mask.

\paragraph{Training}
Given the inference pipeline, we now wish to train the three networks $\mathcal{M}$, $\mathcal{E}$ and $\mathcal{D}$, \textit{without} any supervision.
In order to do this, we introduce two main ideas:\\
\vspace{-1.5em}
\begin{itemize}[leftmargin=*]
    \setlength\itemsep{-.3em}
    \item we can treat the mask created by $\mathcal{M}$ as a latent variable, which with \textit{proper regularization} during training,
    can be discovered automatically (\Section{phaseI});
    \item we can simulate the user input by considering the mask at time $t$ and $t+1$ (\Section{phaseII}).
\end{itemize}
We detail them in the following subsections. %

\subsection{Initial training for mask-based generation}
\label{sec:phaseI}
We first train our method
with the focus of generating high-quality foreground-background segmentation masks, without considering introducing controlability to the generation process.
Doing so requires two main objectives: (1) high-quality estimation of the next image frame $\mathbf{f}^{t+1}$; and (2) proper regularization of the generated mask.
Hence, if we write the two objectives as loss terms $\loss{img}$ and $\loss{mask}$, respectively, the total loss for the first stage training of our method $\loss{total}$ can be written as:
\begin{equation}
    \loss{total}=\loss{img}+\loss{mask}
    .
    \label{eq:loss_total}
\end{equation}
We detail each loss term in the following.

\paragraph{Learning to predict the next frame}
To train our method to estimate the next frame, we follow the VQ-VAE\cite{VQVAE} framework with a Generative Adversarial Network (GAN) loss to encourage realism\cite{taming}.
We therefore write
\begin{equation}
    \loss{img}=\hparam{VQ}\loss{VQ} + \hparam{GAN}\loss{GAN} + \hparam{percept}\loss{percept}
    ,
\end{equation}
where $\hparam{VQ}$, $\hparam{GAN}$, $\hparam{percept}$ are hyperparameters controlling the influence of each loss term.

\subparagraph{$\loss{VQ}$}
Following the original VQ-VAE formulation we write
\begin{equation}
\begin{split}
    \loss{VQ}
    = \left\|\mathbf{f}^{t+1} - \hat{\mathbf{f}}^{t+1} \right\|_1
    & + \left\| \text{sg}\left[\mathcal{E}(\mathbf{f}^t)\right] - \mathbf{z}_\mathsf{q} \right\|^2_2 \\
    & + \left\| \text{sg}\left[\mathbf{z}_\mathsf{q}\right] - \mathcal{E}(\mathbf{f}^t) \right\|^2_2
\end{split}
    ,
\end{equation}
where $\mathbf{f}^{t+1}$ and $\hat{\mathbf{f}}^{t+1}$ are true and estimated next frame, $\text{sg}[\cdot]$ denotes the stop gradient operation, and $\mathbf{z}_\mathsf{q}$ is the quantized latent variable of the VQ-VAE.
Note that we use the $\ell_1$ norm, instead of the $\ell_2$, for the reconstruction part of the loss, as we emprically found it to be more stable in training.

\subparagraph{$\loss{GAN}$}
We train a discriminator $\mathcal{C}$ with the architecture from \cite{pix2pix} and aim to improve the generation quality.
We therefore write
\begin{equation}
    \loss{GAN} = \log\left(\mathcal{C}\left(\mathbf{f}^{t+1}\right)\right) + \log\left(1-\mathcal{C}\left(\hat{\mathbf{f}}^{t+1}\right)\right)
    .
\end{equation}
We note that for the hyperparameter for this loss $\hparam{GAN}$, we follow \cite{taming} and apply a dynamic weighing strategy, which stabilizes training.

\subparagraph{$\loss{percept}$}
We use a pretrained VGG-16 network~\cite{VGG} to extract deep features and compute the perceptual loss.
Denoting the deep feature extraction process as $\mathcal{V}$ we write
\begin{equation}
    \loss{percept} = \left\|\mathcal{V}(\hat{\mathbf{f}}^{t+1}) - \mathcal{V}(\mathbf{f}^{t+1}) \right\|_2
    .
\end{equation}

\paragraph{Regularizing the mask}
To train our method to generate proper masks without any supervision, we regularize the mask with three losses:
(1) $\loss{bg}$ -- the contents of the background should not change;
(2) $\loss{fg}$ -- there should be as least amount of foreground as possible  since classifying all pixels as foreground provides a trivial solution for $\loss{bg}$; and
(3) $\loss{bin}$ -- the masking should be binary for effective separation.
We therefore write the mask regularization loss $\loss{mask}$ as
\begin{equation}
\label{eq:mask_loss}
    \loss{mask}=\hparam{bg}\loss{bg}+\hparam{fg}\loss{fg}+\hparam{bin}\loss{bin}
    ,
\end{equation}
where $\hparam{bg}$, $\hparam{fg}$, and $\hparam{bin}$ are the hyperparameters controlling how much each loss term affects the mask regularization.

\subparagraph{$\loss{bg}$}
We aim to ensure that the mask correctly identifies the background, \ie, non-moving parts of the scene. Hence, we simply define it as the amount of change in the masked out (background) region between two consecutive frames.
We write
\begin{equation}
\begin{split}
    \loss{bg}&=
    \left\|
        (1-\mathbf{m}^t)\odot\mathbf{f}^t - (1-\mathbf{m}^t)\odot\mathbf{f}^{t+1}
    \right\|_1
\end{split}
    ,
    \label{eq:loss_bg}
\end{equation}
where $\odot$ denotes the elementwise multiplication.
Note that we define this loss using the $\ell_1$ norm, as changes in the scene are not strictly restricted to the mask -- \eg shadows of moving objects can occur in the background, or other scenic changes, such as a global illumination change can happen -- and the $\ell_1$ norm leaves room for the method to incorporate these edits if necessary.

\subparagraph{$\loss{fg}$}
As mentioned earlier, $\loss{bg}$ alone, leaves room for a trivial solution---assigning $\mathbf{m}^t{=}1$ results in $\loss{bg}{=}0$ regardless of the values of $\mathbf{f}^t$ and $\mathbf{f}^{t+1}$.
This could be avoided by enforcing an additional loss term that penalizes having too many foreground pixels, but a na\"ive regularization is not sufficient, as the amount of the actual foreground pixels to be edited may drastically differ from one frame to another -- \eg, robot arm moving close to the camera vs. further away.

We thus propose to regularize based on the amount of evident change between $\mathbf{f}^t$ and $\mathbf{f}^{t+1}$, approximated using simple background subtraction.
Specifically, for a pixel index $i$, $j$, if we denote whether the pixel changed between the two consecutive frames $\mathbf{f}^t$ and $\mathbf{f}^{t+1}$ as
\begin{align}
    \boldsymbol{\mu}^t_{ij}=
    \begin{cases}
        1, & \text{if } \left\|\mathbf{f}^{t+1}_{ij}-\mathbf{f}^{t}_{ij}\right\|_1 > \tau \\
        0, & \text{otherwise}
    \end{cases}
    ,
\end{align}
where $\tau$ is a threshold for controlling the sensitivity, we can use the average of $\boldsymbol{\mu}_{ij}^t$ as a rough standard
for how much of the pixels should be foreground, dynamically for each consecutive frame.
We thus write
\begin{equation}
    \loss{fg} = \max\left\{0, \left\|\mathbb{E}_{ij}\left[{\mathbf{m}}_{ij}^t\right]-\mathbb{E}_{ij}\left[{\boldsymbol{\mu}}_{ij}^t\right]\right\|_1\right\}
    ,
    \label{eq:loss_fg}
\end{equation}
where $\mathbb{E}_{ij}$ is the expectation over pixels.

We note that the balance between the hyperparameter settings related to $\loss{fg}$ and $\loss{bg}$ is important since they govern how the loss behaves---\eg a wider mask that covers all potential changes in the scene, or a tight mask that only focuses on the actual changing locations.
We empirically set the ratio between $\lambda_{bg}$ and $\lambda_{fg}$ to be 100:1, and we gradually decrease this ratio in training for faster convergence.

\subparagraph{$\loss{bin}$}
Finally, there is one last loophole for the network to cheat its way through the two mask regularizors -- by producing intermediate values in the range $[0, 1]$.
In fact, we found in our early experiments that this \textit{soft mask} allows the deep network to encode information about the image, \ie, $\mathbf{f}^{t+1}$, thus being able to fully reconstruct the original image from mask alone, and any modification on the mask -- \eg, position shifts -- results in a global appearance change of the generated image, which is undesirable.
Therefore, we encourage the mask to be binary as in \cite{Bielski_NeurIPS2019}:
\begin{equation}
    \loss{bin} = \min\{\mathbf{m}^t, (1-\mathbf{m}^t)\}
    .
    \label{eq:loss_bin}
\end{equation}
We then, at test time, convert these pseudo-binary masks into hard ones by thresholding with 0.5.

\subsection{Fine-tuning for controllability}
\label{sec:phaseII}
We now discuss the second stage of our training setup, where we shift our focus to imbue our method with controllability.
While the model trained in \Section{phaseI} is a generative model conditioned on the latent mask $\mathbf{m}^t$, it cannot immediately be used with any arbitrary mask -- the generator $\mathcal{G}$ would expect a mask that aligns perfectly with $\mathbf{f}^t$, whereas our user controlled mask $\mathcal{T}\left(\mathbf{m}^t, \boldsymbol{\theta}^t\right)$ will not.
In other words, we need a way to simulate user input, in terms of mask modifications, and incorporate it into the training.

We therefore turn our attention to the fact that the masks between two consecutive frames $\mathbf{m}^t$ and $\mathbf{m}^{t+1}$ are not too different, and that $\mathbf{m}^{t+1}$ can be seen as user modified version of $\mathbf{m}^t$ that specifies how the foreground should move, that is,
\begin{equation}
    \mathbf{m}^{t+1} \approx \mathcal{T}\left(\mathbf{m}^t,\boldsymbol{\theta}^t\right)
    .
\end{equation}
Under this assumption, we can then find the pseudo user control at time $t$ using the following optimization procedure \cite{Jadeberg15,Jiang19}:
\begin{equation}
    \hat{\boldsymbol{\theta}}^t \equiv \argmin_{\boldsymbol{\theta}}\left\|
    \mathbf{m}^{t+1} - \mathcal{T}\left(\mathbf{m}^t,\boldsymbol{\theta}\right)
    \right\|
    .
    \label{eq:optimization}
\end{equation}
Now, with $\hat{\boldsymbol{\theta}}^t$, we can fine-tune our network $\mathcal{G}$ to create the next frame, which results in controllable video generation.
Denoting the binarization operation as $\lfloor\cdot\rfloor_{0.5}$, instead of \Equation{gennet}, we write
\begin{equation}
    \tilde{\mathbf{f}}^{t+1}=\mathcal{G}\left(\mathbf{f}^t, \left\lfloor
        \mathcal{T}\left(\mathbf{m}^t, \hat{\boldsymbol{\theta}}^t\right)
    \right\rfloor_{0.5}\right)
    .
    \label{eq:next_frame}
\end{equation}
We then use $\tilde{\mathbf{f}}^{t+1}$ in our loss functions to fine-tune. %

One noteworthy aspect of this second stage training is that, because we binarize the mask, no gradient flows through to $\mathcal{M}$.
We found this to be important, as allowing \textit{any} softness in the mask resulted in the image information leaking through the mask, resulting in non-controllabilty -- \eg, shifting the mask resulted in a shift of the entire scene.
This also leads to $\loss{bg}$, $\loss{fg}$, and $\loss{bin}$ not affecting training.
While the latter two can be dropped since they are purely on how the mask network $\mathcal{M}$ behaves, completely dropping $\loss{bg}$ now has the danger of the generated image ignoring the mask.
Hence, we replace $\mathbf{f}^t$ in \Equation{loss_bg} with $\tilde{\mathbf{f}}^{t+1}$, so that the generated image still obeys the mask conditioning.
Hence, for the second stage training, instead of $\loss{bg}$, we utilize $\loss{bg}'$ where
\begin{equation}
\begin{split}
    \loss{bg}'&=
    \left\|
        (1-\mathbf{m}^t)\odot\tilde{\mathbf{f}}^{t+1} - (1-\mathbf{m}^t)\odot\mathbf{f}^{t+1}
    \right\|_1
\end{split}
    ,
    \label{eq:loss_bg2}
\end{equation}
which now enforces our fine-tuned generator $\mathcal{G}$ to still obey the provided mask.

\section{Acknowledgements}
\vspace{-0.02in}
This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC and an NSERC Discovery and Discovery Accelerator Grants. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute \url{www.vectorinstitute.ai/#partners}. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition awards of the two PIs. We would also like to thank Willi Menapace for his help in answering our questions and helping with fair comparisons to \cite{playable}.

redpajama_set_name: RedPajamaArXiv


is a leading supplier of specialty metals and hard to find steel products. We have been servicing machine shops, pump and valve manufacturers and repair centers for over 30 years. Our products have been used in chemical and petrochemical manufacturing, turbine manufacturing, power generation, oil field equipment, ship building, pulp and paper mills, and ring gasket manufacturing.
Read about the centrifugal casting process and learn about the benefits.
We specialize in stainless steel and alloy centrifugal castings and solids in many different materials, view our full products list.
Whether you are looking for something that is in our stock or a hard to find material or size Let us know what you are looking for.
At Texcellent, we are committed to customer service. We understand the demands on your business, and we pride ourselves on delivering the high quality materials that your job requires, quickly. At Texcellent, we are dedicated to supplying your needs, right the first time, on time.