This is really brilliant, literally jaw-dropping. I was really blown away by the idea and construction of the detector itself and this application is astounding.
My favourite story about the construction of IceCube relates to the drilling of the holes to deploy the detector. They used two types of drill, which melt the firn (compacted snow) on top and ice lower down. One is a sort of conical heat exchanger, the other circulates water through a Big water heater on the surface.
In the early days of South Pole Station (late 1950s), supplies were airdropped in - the cargo planes of that era couldn't land on the snow and going overland was very difficult. Many of the parachutes wouldn't open properly, and so lots of supplies (including a bulldozer) and parachutes wound up buried in the snow. South Pole slowly accumulates snow, so that debris has gotten deeper and deeper below the surface over time.
No mentions about finding meat here, but you might be interested in this story at IEEE Spectrum about the construction of IceCube from 2011 .
You may like KM3NeT then (http://www.km3net.org/). The goal is to build a cubic kilometer spanning instrument at the bottom of the Mediterranean.
I always thought that if an alien civilization is trying to communicate with earth, they would do it via neutrinos and not radio waves, because if you know where you are communicating to, you don't need to worry about signal attenuation as much as with EM waves.
Supplementary Figure 6 is pretty cool! There seems to be a lot of strange things happening at the mantle-crust boundary that will take some further study to understand.
All said, this is a MAJOR advance in our understanding of the Earth and the science thereof. This technique should allow for some pretty amazing things in the future.
Would it be feasible to put a source and detector in orbit so we could get high resolution CT images of the interior of the earth? IIRC the detector used here is very large, but there are smaller ones and a directed source would go a long way towards making the detection simpler.
Neutrino detection generally requires large volumes and large masses (for very large values of "large"), neither of which are as yet feasible for space based systems. The IceCube neutrino observatory, for example, makes use of a volume of ice that is on the order of a cubic kilometer (over 900 million tonnes). Suffice it to say, building cubic kilometer solid structures in space that weigh hundreds of millions of tonnes is outside of our engineering capabilities at present. However, within the next several decades this might be feasible.
That's also understating the size. While the final detection is happening in the ice, the column of air above it is also important.
Could we not just land a detector on the far side of the moon. Thats pretty large no?
It is but you need both a big hunk of mass and a way to detect the neutrino interactions inside it. Ice Cube is shot through with detectors. One of the OG detectors measured the particular isotope created during interactions.
None of these are much of a practical option with the moon.
You need to have the reaction mass to make use of. Most neutrino detectors use special chemicals in custom tanks, IceCube uses antarctic ice. You would need not just mass but ice or water for a neutrino detector, ideally highly compressed optically clear ice. The only possible candidate in the Solar System that fits that bill at present is Europa. Though it's possible that similar conditions might also exist on Mars (we know there are sub-surface glaciers, but we don't know how deep they go or how pure the ice is. And potentially some of the large sub-surface oceans (on Ganymede, Enceladus, etc.) might work too but we know even less about their properties.
Not all detectors were optical so if you found a moon made of, say, dry cleaning fluid, you'd still be good.
Each detector has a limited range. And only works well in certain mediums, generally water or water ice. Even if the Moon was solid ice, we would need to surround it with detectors, as well as drill a huge number of holes all the way through it and place detectors (probably hundreds of thousands or millions) in each hole.
I had this idea back in my undergrad when my professor first told me about neutrinos (which was his forte). The harder I looked the more infeasible it seemed. The big problem, is creating a detector. The reason this is a big problem is because neutrinos are so small and so neutral. How small? An electron is 511keV. A neutrino is <= 0.21eV. That's 7 orders of magnitude! And on top of that, they have a neutral charge. There is no electric dipole moment either. So how does it interact? Basically by striking another particle. And remember that atoms are basically empty space. So chances of that are slim.
The former also comes with some problems. You'll find more about this by asking why detectors are so far underground. And there's a big connection there to generating the resolution you'd need to map the interior of the Earth.
An in space detector is (at least to my understanding) not that easy, as you need the surface of the Earth to interact with the neutrinos. These interactions result in muons, which allow to detect that there was a neutrino in the first place. Also, there is a pretty big source for neutrinos pretty close to Earth, but Earth is in its orbit, as opposed to the other way around.
The IceCube neutrino detector consists of about a cubic kilometer of ice. You're not going to put one of those in orbit.
But it would make a cool joint project for NASA and the NSF to fund. Capture a comet big enough that you could mine enough water to make a cubic kilometer sized detector in Earth orbit.
Then again, it would look spectacular in the sky!
There are other types of neutrino detectors. Not that it really makes the problem any easier to solve, but this isn't the only way.
That's what I was thinking. They have sent neutrinos from one lab to another with a man-made source and a detector smaller than 1km^3. The idea is to put a strong source in orbit and aim it at an orbiting detector. Over time this configuration would be used to scan the entire planet. Still a huge challenge.
That's essentially the main idea most people come to. Using a source so you can concentrate on specific ones and not get distracted by solar, cosmic, or terrestrial neutrinos. Or you could just use terrestrial ones. But still, that detector is the hardest part to figure out.
You might be able to tow it in from elsewhere using a light sail, however. Beyond our current technology, but might be doable by 2100, or sooner with a fortuitous comet trajectory.
And once it is in orbit, how do you propose stationkeeping against exospheric drag? Preferably a solution that will remain reliable for 10^n (n>>3) years.
A cubic kilometer of ice is 9.167e11 kg.
Also, just keep the orbit above 800km, and the light sail will still be able to counteract its own drag.
If the orbit gets too low, turn the ice into rocket fuel and use it to refuel ships boosting into higher orbits.
I'm pretty sure that a cubic kilometer of ice in orbit would have more value in other purposes than as a neutrino detector!
Sure, but as long as it's up there, might as well use it in every way possible.
The muons are produced in the upper atmosphere (from cosmic rays interacting with the atmosphere). You can't do this in space. Not too mention the size of a neutrino detector, although you can use the e.g. regolith of the moon as a neutrino detector (via the Askaryan effect). In fact there are experiments pointing radio telescopes at the moon to look for ultra-high-energy neutrino interactions. My not-so-polite term for them is lunatics :).
Nice but I'm still left wondering what the J/bit of neutrino transmission would be. Wouldn't it be cool if we could replace the long fiber optics stretching around the globe with direct-through neutrino stations?
Given the difficulty in neutrino detection, I seriously doubt it would replace fat links. You could gain quite a bit of advantage when it comes to trading though using a low bandwidth link.
https://what-if.xkcd.com/73/ gives a fairly decent summary of the level of interaction of neutrinos.
Frankly this idea has been repeated every few years since the invention of tomography 40 years ago. Hopefully someone will actually do it one of these years.
In theory, would IceCube be able to detect neutrinos generated by the same mechanism in the atmosphere of Venus?
In theory, yes. In practice, probably very difficult. They did detect a deficit in neutrinos from the direction of the moon though .
Could this technique be useful on other planets?
Looks like my prediction will come true.
> Neutrino detection will revolutionize oil and other natural resource discovery.
To be fair, it probably will. But the ability to do it in a cheap enough manner is certainly far enough away that the value of those resources will likely be much lower than they are today.
Accurate, cheap, and small neutrino detectors would revolutionize any industry dependent upon locating things within... well... anything. But that statement is as meaningful (if not less) as "faster computers will revolutionize the automotive industry."


text:
Juvenile Misses Daughter's Funeral To Avoid Media Attention: Report
Four-year-old Jelani, her mother and sister laid to rest in South Carolina on Saturday.
Jayson Rodriguez 03/10/2008
Juvenile's 4-year-old daughter, Jelani Deleston, her mother and her older sister were laid to rest in James Island, South Carolina, on Saturday, just over a week after the three were brutally murdered in their Georgia home.
According to several media outlets, Juvenile reportedly did not attend the funeral services. TMZ, citing an unnamed spokesperson for the rapper, reported that he declined to attend because he did not want to attract media to the gathering.
Jelani, Joy Deleston, 39, and Micaiah, 11, were shot and killed on February 28, allegedly by Deleston's 17-year-old son, Anthony Tyrone Terrell Jr. Terrell has been arrested and charged with three counts of murder and three counts of aggravated assault. He is currently being held in a DeKalb County prison. Because of his age, prosecutors cannot seek the death penalty against him.
Juvenile (real name: Terius Gray) has kept a low profile during the ordeal, with the only public comment being a statement from his manager saying that the rapper was "shocked and devastated" to learn of his daughter's death. The extent of Juvenile's relationship with Jelani or her mother, a sheriff's deputy, is unclear.
Deleston brought a paternity lawsuit against Juvenile in 2004 claiming the rapper was Jelani's father. Attorney Randy Kessler, who represented Juvenile in the proceedings, told The Associated Press that the case was resolved peacefully by consent order in 2006, with both parties agreeing that Juvenile was the father. Juvenile had been making regular child-support payments, his manager told the AP.

redpajama_set_name: RedPajamaCommonCrawl


83 Tranquility Lane, Destin, FL 32541 (MLS # 811009) | Real Estate from Navarre to Panama City!
This Victorian Home is located in the gated community of Destiny East. The main house has 4 bedrooms and 4 bathrooms, the carriage house in the back home includes 2 bedrooms and 1 bath. The living room is octagon shape with a 20 ft ceiling and gas fireplace with black marble hearth. The kitchen is spacious with solid granite through out - plus a breakfast nook. Master Bath has an extra large Jacuzzi tub with extra jets and two walk in closets. The garage is extra large with a workshop and storage space above. Parking for multiple cars with a long paver driveway. Owner is a licensed Florida Real Estate Agent.


text:
Pallet & Packaging…
Timber Components
Hardwoods & specialist…
Firewood, Sawdust,…
Sawmill Processes
Co-products
Sourcing Requirements
Harvesting Enquiry
Wood - the ultimate…
Sustainable forest…
A&J Scott Policy
Grown In Britain
A&J Scott Launches Fencing Stock Centre
Posted: 4th February 2011
One of the UK's leading independent homegrown sawmills is set to launch a new operation to provide improved service and flexibility to fencing customers.
A & J Scott Ltd, based near Alnwick in Northumberland, supplies sawn timber products for customers in the fencing, landscaping, outdoor and garden products, DIY, pallet and packaging and other industrial sectors. The company has built up a reputation for quality and service over 50 years, cutting timber to order for a wide variety of customers.
Now though the company is to launch a fencing stock centre to provide key products from stock, with a quick turnaround to provide a better service to customers seeking to work more flexibly and reduce their own stock holdings. The company also expects to find new customers who require smaller volumes than its cut to order service provides.
Robert Scott, who took over as Managing Director from his father Andrew earlier this year, said, "For years now we have been proud of the quality of products we have supplied and the unrivalled service standards we have within the company. With the fencing stock centre in place we will be able to provide even better service to a wider range of customers."
The company is currently building up its stock of fencing products and expects the stock centre to be fully operational to enter the new year, with stocks built further to meet rising demand during the peak season. For more information call 01668 217 288.
2011 Copyright A&J Scott Ltd,
Station Sawmills, Wooperton, Alnwick, Northumberland, NE66 4XW
Company Registered No. 01087941
Website by Presence Multimedia


During 2010 and 2011, many of us worked long hours to help serve up some justice to the many animals killed and mistreated by Brett and Alexis Ingraham. Read more about the Ingrahams here.
The Ingrahams were found guilty of animal cruelty. But did they learn? Some say not.
"Incorrigible" means unable to be corrected or reformed. That's what Dr. Janell Tirrell, of Third Coast Equine, called folks like the Ingrahams when she met with fellow advocates to help develop what would eventually be known as the Maine Equine Welfare Alliance.
Janet Tuttle, owner of Rockin' T Equine Rescue, used her usual blue-collar bluntness to say the same thing of horse owners who fail to embrace the real meaning of ownership: Of folks like these, "you cannot learn stupid," she told me.
Meanwhile, a group in Houlton nowadays has been trying to effect change at Jessica York's farm. Check out this report by WGME.
Since I moved to Utah from Maine, I've bore sad witness to a few horse neglect incidents, big and small. The most recent concerned a neighbor who took off for four days of vacation without bothering to have anyone tend to his equines. Sure, he piled hay in their racks and filled the tub with water. But after 36 hours, they had eaten all the hay. What water they had was slimy and warm (since it sat in an uncleaned, algae-filled tub).
Maine Equine Welfare Allianchours, they had eaten all the hay and the water had grown slimy (since he had not been cleaning the tub, apparently).
Did I mention temperatures were consistently in the 90s?
By Day Three, a group of concerned neighbors called animal welfare.
By Day Four, the horses finally had hay and fresh water. Did the owner learn anything when he had to chat with the animal welfare officer? I wonder.
That same week, I parked next to a big, red Cadillac with two dogs in it. The windows were rolled down about an inch and a half. The temperature was 95 degrees.
I called the sheriff. He called an animal control officer who measured the temperature in the car at 115 degrees. They broke into the car, saved the dogs, and gave a citation to the owner. Did that owner learn anything? I wonder, sometimes, if the world is not full of incorrigibles.
So, this week and every week, my hat is off to all those working to educate folks about this great privilege and responsibility we have: Ownership. My Thank You list includes vets like Tirrell, Dr. Rachel Flaherty and South Mountain Equine, groups like the MSSPA. And all those quiet workers, neighbors, friends, and family who help nurture good horsemanship and husbandry with those who might not know better.
And for all those out there fearlessly advocating for animals: keep up the good work!
Posted in Equine Welfare, Maine.


Once I am in the woods, I try to find rabbit tracks in the snow and then work my way to the heavily traveled leads. I then follow the trail and attempt to find their hiding or feeding spots, often found near young trees that are budding. Very low cedar, pine trees are a very good place to look or even under fallen dead trees that create hiding pockets. There you should find urine stains, green and brown droppings. If they are hiding out and you know what to look for, examples are yellow stained paws or the monocular shiny eyes and black stain ear tips. You may harvest unless you flush them out in which case they may circle, so stay where you are.


text:
\section{Introduction}

Trailers are short videos used for promoting movies and are often
critical to commercial success. While their core function is to market
the film to a range of audiences, trailers are also a form of
persuasive art and promotional narrative, designed to make viewers
want to see the movie. Even though the making of trailers is
considered an artistic endeavor, the film industry has developed
strategies guiding trailer construction. According to one school of
thought, trailers must exhibit a narrative structure, consisting of three acts\footnote{\href{https://www.studiobinder.com/blog/how-to-make-a-movie-trailer}{https://www.studiobinder.com/blog/how-to-make-a-movie-trailer}}.
The first act establishes the
characters and setup of the story, the second act introduces the main
conflict, and the third act raises the stakes and provides teasers
from the ending. Another school of thought is more concerned with the
  mood of the trailer as defined by the ups and downs of the story\footnote{\href{https://www.derek-lieu.com/blog/2017/9/10/the-matrix-is-a-trailer-editors-dream}{https://www.derek-lieu.com/blog/2017/9/10/the-matrix-is-a-trailer-editors-dream}}.
According to this approach, trailers should have medium intensity at
first in order to captivate viewers, followed by low intensity for
delivering key information about the story, and then progressively
increasing intensity until reaching a climax at the end of the
trailer.

To create trailers \emph{automatically}, we need to perform low-level
tasks such as person identification, action recognition, and sentiment
prediction, but also more high-level ones such as understanding
connections between events and their causality, as well as drawing
inferences about the characters and their actions.  Given the
complexity of the task, \emph{directly} learning all this knowledge
from movie-trailer pairs would require many thousands of examples,
whose processing and annotation would be a challenge. It
is thus not surprising that previous approaches to automatic trailer
generation~\cite{irie2010automatic,smeaton2006automatically,wang2020learning}
have solely focused on audiovisual features.

Inspired by the creative process of human editors, we adopt a
bottom-up approach to trailer generation, which we decompose into two
orthogonal, simpler and well-defined subtasks. The first is the
identification of narrative structure, i.e.,~retrieving the most
important events of the movie. A commonly adopted theory in
screenwriting
\cite{cutting2016narrative,Hauge:2017,thompson1999storytelling}
suggests that there are five types of key events in a movie's plot,
known as turning points (TPs; see their definitions in
Figure~\ref{fig:TPs}).  The second subtask is sentiment prediction,
which we view as an approximation of flow of intensity between shots
and the emotions evoked.

\begin{figure}[t]
\centering
\begin{small}
    \begin{tabular}{@{}l@{~}p{4.7cm}@{}} \hline
      \multicolumn{1}{@{}l}{1. Opportunity} &
      Introductory event that occurs after presentation of setting and
      background of main characters. \\ \hline
      \multicolumn{1}{@{}l}{2. Change of Plans}&
      Main goal of story is defined; action begins
      to increase. \\\hline
      \multicolumn{1}{@{}l}{3. Point of No Return} &
      Event that pushes the characters to
      fully commit to their goal. \\ \hline
      \multicolumn{1}{@{}l}{4. Major Setback} & Event where everything falls apart,
      temporarily or permanently.  \\ \hline
      \multicolumn{1}{@{}l}{5. Climax} & Final event of the main story,
      moment of resolution. \\\hline
\end{tabular}
\end{small}
\vspace{-1em}
\caption{\label{fig:TPs} Turning points and
  their definitions.}
\vspace{-1.5em}
\end{figure}

We generate \textit{proposal} trailers following an unsupervised
graph-based approach.  We model movies as graphs whose nodes are shots
and whose edges denote important semantic connections between shots
(see Figure~\ref{fig:intro}). In addition, nodes bear labels denoting
whether they are key events (i.e.,~TPs) and scores signaling sentiment
intensity (positive or negative). Our algorithm traverses this movie
graph to create trailer sequences. These could be used as proposals to
be reviewed and modified by a human editor.

Both the tasks of TP identification and sentiment prediction stand to
benefit from a lower-level understanding of movie content. Indeed, we
could employ off-the-shelf modules for identifying characters and
places, recognizing actions, and localizing semantic units. However,
such approaches substantially increase pre-processing time and memory
requirements during training and inference and suffer from error
propagation. Instead, we propose a contrastive learning regime, where
we take advantage of screenplays as \emph{privileged information},
i.e., information available at \emph{training time} only. Screenplays
reveal how the movie is segmented into scenes,
who the characters are, when and who they are speaking to, where they
are and what they are doing (i.e., ``scene headings'' explain where
the action takes place while ``action lines'' describe what the camera
sees).  Specifically, we build two individual networks, a
\textit{textual} network based on screenplays and a
\textit{multimodal} one based on video, and train them jointly using
auxiliary contrastive losses.  The textual network can additionally be
pretrained on large collections of screenplays via self-supervised
learning, without having to collect and process the corresponding
movies.  Experimental results show that this contrastive training
approach is beneficial, leading to trailers which are judged favorably
by humans in terms of their content and attractiveness.

\section{Related Work}

Previous approaches to \textbf{movie understanding} have mainly
focused on isolated video clips, and tasks such as the alignment
between movie scenes and book chapters \cite{tapaswi2015book2movie},
question answering \cite{tapaswi2016movieqa}, video captioning for
movie shots \cite{rohrbach2015dataset}, and text-to-video retrieval
\cite{bain2020condensed}.
Recent work
\cite{papalampidi2019movie,papalampidi2020screenplay,papalampidi2020movie}
attempts to identify high-level narrative structure and summarize
entire TV episodes and movies focusing exclusively on the textual
modality (i.e.,~screenplays).

Existing approaches to \textbf{trailer generation} exploit superficial
audiovisual features, such as background music or visual changes
between sequential
shots~\cite{irie2010automatic,smeaton2006automatically}. Other work
creates ``attractive'' trailers with a graph-based model for shot
selection \cite{xu2015trailer} or uses a human in the loop in
conjunction with a model trained on horror movies via audiovisual
sentiment analysis \cite{smith2017harnessing}. The Trailer Moment
Detection Dataset \cite{wang2020learning} consists of full-length
movies paired with official trailers and annotations for key moments,
but it is not publicly available and does not include screenplays.

\textbf{Knowledge distillation} \cite{ba2014deep,hinton2015distilling}
was originally proposed for distilling information from a larger
teacher model to a smaller student one. Generalized distillation
\cite{lopez2015unifying} provides a framework for using privileged
information, i.e.,~information which is available at training time
only.  Most related to our work is the use of different modalities or
views of the same content \cite{miech2019howto100m,miech2020end},
e.g., transcribed narrations to learn visual representations in
instructional videos. We leverage screenplays as a source of
privileged information and distill knowledge about events, characters,
and scenes in a film, which we subsequently exploit for identifying
trailer-worthy shots in video.

\section{Problem Formulation} \label{sec:problem_formulation}

Trailer generation requires the selection of $L$~shots from a
full-length movie of~$M$ shots ($L \ll M$).  Movies present complex
stories that may contain distinct subplots or events that unfold
non-linearly, while redundant events, called ``fillers'' enrich the main
story. Hence, we cannot assume that consecutive shots are necessarily
semantically related. To better explore relations between events, we
represent movies as graphs \cite{papalampidi2020movie}. Let
$\mathcal{G}=(\mathcal{V}, \mathcal{E})$ denote a graph where vertices
$\mathcal{V}$ are shots and edges~$\mathcal{E}$ represent
their semantic similarity.  We further consider the original temporal
order of shots in $\mathcal{G}$ by only allowing directed edges from
previous to future shots. $\mathcal{G}$~is described by an
upper-triangular transition matrix $\mathcal{T}$, which records the
probability of transitioning from shot~$i$ to every future shot~$j$.

Within~$\mathcal{G}$, we assume that \emph{some} shots describe key
events in the movie (thick circles in Figure~\ref{fig:intro}) while
\emph{all} shots have a sentiment (positive or negative), whose
intensity is denoted by a score (shades of green/red in
Figure~\ref{fig:intro}).  We propose an algorithm for
traversing~$\mathcal{G}$ and selecting sequences of trailer shots. In
the following, we first describe this algorithm
(Section~\ref{sec:graph_traversal}) and then discuss how the
graph~$\mathcal{G}$ is learned and key events are detected via TP
identification \cite{papalampidi2019movie}
(Section~\ref{sec:tp_identification}). Finally, we also explain how
shot-based sentiment scores are predicted
(Section~\ref{sec:sentiment_prediction}).

\subsection{Movie Graph Traversal} \label{sec:graph_traversal}

\begin{algorithm}[t]
\caption{Graph traversal: Retrieve trailer path}\label{algorithm}
\begin{small}
\textbf{Input:} shot-level graph $\mathcal{G}$, sets of TP shots,
sentiment scores for all shots \\
\textbf{Output:} proposal trailer path (Path)
\begin{algorithmic}[1]
\Procedure{\textsc{Graph\-Trailer}\xspace}{}
\State {\small Path $\gets \emptyset$,  budget $\gets L$,  TPs\_id $\gets 0$,  flow $\gets \emptyset$}

\State events $\gets$ {\small$[\mathcal{TP}_1, \mathcal{TP}_2, \mathcal{TP}_3, \mathcal{TP}_4, \mathcal{TP}_5]$}

\State i $\gets$ sample({\small$\mathcal{TP}_1$})

\State add i to Path

\State next\_TP $\gets$ events[TPs\_id]

\While {budget $> 0$ $\And$ TPs\_id $< 5$}

\State next\_node $\vcentcolon= \operatornamewithlimits{argmax}\limits_{j \in \mathbb{N}_{i}}(s_{ij})$ \Comment{Eq.~\ref{eq:score}}

\State add next\_node to Path

\State add sentiment(next\_node) to flow

\State i $\gets$ next\_node

\State budget++
\If {i $\in$ next\_TP $\cup$ $\mathcal{N}_{next\_TP}$}
\State {\small TPs\_id++, \, next\_TP $\gets$ events[TPs\_id]}

\EndIf

\EndWhile

\Return Path

\EndProcedure
\end{algorithmic}
\end{small}
\end{algorithm}

Algorithm~\ref{algorithm} retrieves trailer sequences by performing
random walks in graph~$\mathcal{G}$. We start by selecting a node
identified as the first TP (i.e.,~Opportunity, see
Figure~\ref{fig:TPs}). Note that TPs extend over $C$~shots and as a
result, our algorithm can produce $C$~different paths as proposal
trailers (see line~4 in Algorithm~\ref{algorithm}).  Given node~$i$,
we decide where to go next by considering its $K$ immediate neighbors
$\mathcal{N}_i$. We select node~$j$ from $\mathcal{N}_i$ as the
$k^\mathit{th}$ shot $n_k$ to add to the path based on the following
criteria: (1)~normalized probability of transition $e_{ij}$ from $i$
to $j$ based on matrix $\mathcal{T}$
(i.e.,~\textit{semantic similarity} between shots), (2)~normalized
distance $t_{ij}$ between shots $i$ and $j$ in the movie
(i.e.,~\textit{temporal proximity}), (3)~normalized shortest path from
node $j$ to next major event $d_{i,\mathcal{TP}}$
(i.e.,~\textit{relevance to the storyline}), and (4)~sentiment
difference $p_{ij}$ from $i$ to $j$ based on the \textit{desired
  sentiment flow} $f_k$ at the $k^\mathit{th}$ step in the path (see
Section~\ref{fig:intro} and Appendix):
\begin{gather}
n_k =
\operatornamewithlimits{argmax}\limits_{j\in\mathcal{N}_i}s_{ij} \\
    s_{ij} = \lambda_1e_{ij} - \lambda_2t_{ij} - \lambda_3d_{i,\mathcal{TP}} -\lambda_4|p_{ij}-f_k| \label{eq:score}
\end{gather}
where $\lambda_1, \lambda_2, \lambda_3, \lambda_4$ are hyperparameters
used to combine the different criteria (tuned on the development set
based on ground-truth trailers).
Note that these criteria are
interpretable and can be easily altered by a user (e.g., add/delete a
criterion, define a different flow $f$). Our approach could therefore
be used for interactive trailer generation (which we do not pursue
here, but see Appendix for an illustration).

We select $L$ shots in total (depending on a target trailer length)
and retrieve a proposal trailer sequence as depicted in
Figure~\ref{fig:intro} (bold line). At each step, we keep track of the
sentiment flow created and the TPs identified thus far
(lines~10 and~13--14 in Algorithm~\ref{algorithm}, respectively).
 A TP event has been selected for presentation in the trailer if
a shot or its immediate neighbors have been added to the path.

\subsection{TP Identification} \label{sec:tp_identification}

\paragraph{Video-based Network} Let $\mathcal{F}$ denote a movie
consisting of $M$ shots $\mathcal{F}= \{h_1, h_2, \dots, h_M\}$. A
neural network model computes the probability $p(y_{it} |h_i,
\mathcal{F}, \theta_1)$, where~$y_{it}$ is a binary label denoting
whether shot~$h_i$ represents TP~$t \in [1,T]$ and $\theta_1$ are
network parameters. Following screenwriting conventions
\cite{Hauge:2017}, we assume there are five TPs ($T=5$) in a movie
(see Figure~\ref{fig:TPs}).

We learn to identify TPs and the graph~$\mathcal{G}$ that serves as
input to our trailer generation algorithm in tandem. We hypothesize
that the training signal provided by TP labels (i.e.,~the narrative
structure) also encourages exploring meaningful semantic connections
between shots via the graph that is learnt
\cite{papalampidi2020movie}. Each shot~$i$ is represented by~$h_i$, a
nonlinear combination of textual, visual, and acoustic modalities
\mbox{$h_i = f([t_i'; v_i'; a_i'])$}, where $f(\cdot)$ is a
fully-connected layer followed by the ReLU
non-linearity.  We compute textual representation $t_i'$ via the
bi-directional attention flow \cite{seo2016bidirectional,kim2020dense}
between subtitles~$t_i$ and two other modalities, namely audio~$a_i$
and video frames~$v_i$:
\begin{gather}
    S_{t_i, a_i} = a_i^{T}t_i, \quad
    S_{t_i, v_i} = v_i^{T}t_i \\
    a_{i,att} = \operatorname{r}(S_{t_i,a_i})a_i, \quad
    v_{i,att} = \operatorname{r}(S_{t_i,v_i})v_i \\
    t_i' = a_{i,att} + v_{i,att} + t_i
\end{gather}
where $\operatorname{r}$ is the softmax function. The final
audio~$a_i'$ and visual~$v_i'$ representations are obtained
analogously. Before fusion, the modalities are projected to a lower
dimension via a fully-connected linear layer and L2 normalization.

We initially construct a fully-connected graph by computing the
pairwise similarity between all multimodal shot vectors
$h_1, h_2, \dots, h_M$.  In order to avoid dense connections in the
graph, which lead to worse contextualization and computational
overhead, we select a small but variable-length neighborhood per shot
and we sparsify the graph accordingly. During sparsification, we also
constrain the adjacency matrix of the graph to be upper triangular
(i.e.,~allowing only future connections between shots). For more
details about the graph sparsification, see
\cite{papalampidi2020movie} and the Appendix.  We also contextualize
all shots with respect to the whole movie using a transformer
encoder~\cite{vaswani2017attention} and additionally encode each shot
via a one-layer Graph Convolution Network (GCN;
\cite{duvenaud2015convolutional,kearnes2016molecular,kipf2017semi}). Finally,
we combine the global (transformer-based) and local (graph-based)
representations for each shot in order to compute the probability
$p(y_{it} |h_i, \mathcal{F}, \theta_1)$. The network is trained by
minimizing the binary cross-entropy loss. After training, we use the
learned sparse graph $\mathcal{G}$ as input to
Algorithm~\ref{algorithm}.

\begin{figure*}[t]
\begin{tabular}{ll}
\begin{minipage}[t]{5.7cm}
\begin{tabular}{c}
\hspace*{-.2cm}\includegraphics[width=.98\columnwidth]{Graph_traversal.pdf}
\end{tabular}
\vspace{-1ex}
    \caption{\textsc{Graph\-Trailer}\xspace: a movie is a graph whose nodes are shots
      and edges denote  relations between them. Each
      shot is characterized by a sentiment score (green/red shades for
      positive/negative values) and labels describing important events
      (thick circles). Our algorithm performs walks in the graph (bold
      line) to generate proposal trailer sequences.}
    \label{fig:intro}
\end{minipage}
&
\begin{minipage}[t]{11cm}
\begin{tabular}{c}
        \hspace*{-.5cm}\includegraphics[width=1.05\textwidth]{original_contrastive_joint.pdf}
\end{tabular}
\vspace{-1ex}
        \caption{Two networks process different views of the
          movie with different degrees of granularity. The video-based
          network takes as input \textit{multimodal} fine-grained shot
          representations based on the movie's video stream. The
          screenplay-based network processes \textit{textual} scene
          representations which are coarse-grained and based on the
          movie's screenplay.  The networks are trained jointly on TP
          identification with losses enforcing prediction and
          representation consistency between them.}
          \label{fig:dist_model}
\end{minipage}
\vspace{-1em}
\end{tabular}
\end{figure*}

\paragraph{Screenplay-based Network}

We also use a screenplay-based network that has similar architecture to the
video-based network described in the previous section, but it
considers textual information only and produces scene-level TP
predictions (see Figure~\ref{fig:dist_model}).
Let \mbox{$\mathcal{D}=\{s_1, s_2, \dots, s_N\}$} denote a screenplay with
a sequence of~$N$ scenes. The screenplay-based network estimates
\textit{scene-level} probabilities $q(y_{it} |s_i, \mathcal{D},
\theta_2)$, quantifying the extent to which scene~$s_i$ corresponds to
the $t^\mathit{th}$ TP. We represent scenes with a small transformer encoder
which operates over sequences of sentence vectors. As with our
video-based network, we compute contextualized scene representations;
a transformer encoder over the entire screenplay yields global
representations, while local ones are obtained with a one-layer GCN
over a sparse scene-level graph.

The video-based model assumes
access to \emph{shot-level} TP labels. However, the only dataset for
TP identification we are aware of is TRIPOD
\cite{papalampidi2019movie}, which contains \emph{scene-level} labels
based on screenplays. To obtain more fine-grained labels, we project
scene-based annotations to shots following a simple one-to-many
mapping (see Section~\ref{sec:experimental-setup} for details).  Since
our training signal is unavoidably noisy, we hypothesize that access
to screenplays would encourage the video-based model to select shots
which are more representative for each TP. In other words, screenplays
represent privileged knowledge and an implicit supervision signal,
while alleviating the need for additional pre-processing during
inference. Moreover, screenplays provide a wealth of additional
information, e.g., about characters and their roles in a scene, or
their actions and emotions (conveyed by lines describing what the
camera sees). This information might otherwise be difficult to
accurately localize in video. Also, unlabeled text corpora of
screenplays are relatively easy to obtain and can be used to
pre-train our network.

\subsection{Knowledge Distillation}
\label{sec:kd}

We now describe our joint training regime for the two networks which
encapsulate different views of the movie in terms of data streams
(multimodal vs. text-only) and their segmentation into semantic units
(shots vs. scenes).

\paragraph{Prediction Consistency Loss} We aim to enforce some degree
of agreement between the TP predictions of the two networks. For this
reason, we train them jointly and introduce additional constraints in
the loss objective. Similarly to knowledge distillation settings
\cite{ba2014deep,hinton2015distilling}, we utilize the KL divergence
loss between the screenplay-based posterior distributions
$q(y_t|\mathcal{D})$ and the video-based ones $p(y_t|\mathcal{F})$.

While in standard knowledge distillation settings both networks
produce probabilities over the same units, in our case, one network
predicts TPs for shots and the other one for scenes. We obtain
scene-level probabilities $\overline{p(y_t|\mathcal{F})}$ by
aggregating shot-level ones via max pooling and re-normalization.  We
then calculate the prediction consistency loss between the two
networks as:
\begin{gather}
\mathcal{P} =
\frac{1}{T}\sum_{t=1}^{T}\mathcal{D}_{KL}\left(\overline{p(y_t|\mathcal{F})}
  \,\,\middle\|\, q(y_t|\mathcal{D}) \right)
\end{gather}

\paragraph{Representation Consistency Loss} We propose using a second
regularization loss between the two networks in order to also enforce
consistency between the two graph-based representations (i.e.,~over
video shots and screenplay scenes). The purpose of this loss is
twofold: to improve TP predictions for the two networks, as shown in
previous work on contrastive representation learning
\cite{oord2018representation,sun2020contrastive,pan2020spatio}, and
also to help learn more accurate connections between shots (recall
that the shot-based graph serves as input to our trailer generation
algorithm; Section~\ref{sec:graph_traversal}).  In comparison with
screenplay scenes, which describe self-contained events in a movie,
video shots are only a few seconds long and rely on surrounding
context for their meaning. We hypothesize that by enforcing the graph
neighborhood for a shot to preserve semantics similar to the
corresponding screenplay scene, we will encourage the selection of
appropriate neighbors in the shot-based graph.

We again first address the problem of varying granularity in the
representations of the two networks. We compute an aggregated
scene-level representation $\overline{h_j}$ based on shots
$h_i, \dots, h_{i+k}$ via mean pooling and calculate the noise
contrastive estimation (NCE;
\cite{gutmann2010noise,wu2018unsupervised}) loss for the
$j^\mathit{th}$ scene:
\begin{gather}
    \mathcal{R} = - \frac{1}{N} \sum_{j=1}^{N} \log
    \frac{e^{s(\overline{h_j}, s_{j})/\tau}}{e^{s(\overline{h_j},
        s_{j})/\tau} + \sum_{\substack{k=1 \\ k\neq j}}^{N}e^{s(\overline{h_j},
        s_k)/\tau}}  \label{eq:standard_contrastive}
\end{gather}
where $s_j$ is the scene representation calculated by the GCN in the
screenplay-based network, $\overline{h_j}$ is scene representation
calculated by the GCN in the video-based network, $s(\cdot)$ is a
similarity function (we use the scaled dot product between two
vectors), and $\tau$~is a temperature hyperparameter.

\paragraph{Joint Training} Our final joint training objective takes
into account the individual losses~$\mathcal{S}$ and~$\mathcal{V}$ of
the screenplay- and video-based networks (see Appendix for details),
and the two consistency losses $\mathcal{P}$ (for prediction) and
$\mathcal{R}$ (for representation):
\begin{gather}
    \mathcal{L}_{TP} = \mathcal{S} + \mathcal{V} + a\mathcal{P} + b\mathcal{R} \label{eq:loss}
\end{gather}
where $a,b$ are hyperparameters modulating the importance of
prediction vs. representation consistency.
Figure~\ref{fig:dist_model} provides a high-level illustration of our
training regime.

\subsection{Self-supervised Pretraining}
\label{sec:pretraining}
Pretraining aims to learn better scene representations from
screenplays which are more accessible than movie videos (e.g.,~fewer
copyright issues and less computational overhead) in the hope that
this knowledge will transfer to the video-based network via our
consistency losses.

Pretraining takes place on Scriptbase
\cite{gorinski-lapata-2015-movie}, a dataset which consists of
$\sim$1,100 full-length screenplays (approx. 140K~scenes). We adapt, a
self-supervised task, namely Contrastive Predictive Coding (CPC;
\cite{oord2018representation}) to our setting: given a
(contextualized) scene representation, we learn to predict a future
representation in the movie. We consider a context window of several
future scenes, rather than just one. This is an attempt to account for
non-linearities in the movie, which can occur because of unrelated
intervening events and subplots.  Given the representation of an
anchor scene $g_i$, a positive future representation $c_i^{+}$ and a
set of negative examples $\{c_{i1}^{-}, \dots, c_{i(N-1)}^{-}\}$, we
compute the InfoNCE \cite{oord2018representation} loss:
\begin{gather}
    \hspace{-1em}\mathcal{L}_\mathit{self} = - \frac{1}{N} \sum_{j=1}^{N} \log \frac{e^{s(g_i, c_i^{+})/\tau}}{e^{s(g_i, c_i^{+})/\tau} + \sum_{k=1}^{N-1}e^{s(g_i, c_{ik}^{-})/\tau}} \label{eq:info_nce}
\end{gather}
We obtain scene representations~$g_i$ based on $s_i$ provided by the
one-layer GCN. Starting from the current scene, we perform a random
walk of~$k$ steps and compute~$g_i$ from the retrieved path~$p_i$ in
the graph via mean pooling (more details can be found in the Appendix).

\subsection{Sentiment Prediction}
\label{sec:sentiment_prediction}

Finally, our model takes into account how sentiment flows from one
shot to the next. We predict sentiment scores per shot with the same
joint architecture (Section~\ref{sec:kd}) and training regime we use
for TP identification. The video-based network is trained on shots
with sentiment labels (i.e.,~positive, negative, neutral), while the
screenplay-based network is trained on scenes with sentiment labels
(Section~\ref{sec:experimental-setup} explains how the labels are
obtained). After training, we predict a probability distribution over
sentiment labels per shot to capture sentiment flow and discriminate
between high- and low-intensity shots (see Appendix for details).

\section{Experimental Setup}
\label{sec:experimental-setup}

\paragraph{Datasets}
Our model was trained on TRIPOD$\bigoplus$, an expanded version of the
TRIPOD dataset~\cite{papalampidi2019movie,papalampidi2020movie} which
contains 122 screenplays with silver-standard TP annotations
(scene-level)\footnote{\href{https://github.com/ppapalampidi/TRIPOD}{https://github.com/ppapalampidi/TRIPOD}} and the corresponding videos.\footnote{\href{https://datashare.ed.ac.uk/handle/10283/3819}{https://datashare.ed.ac.uk/handle/10283/3819}} For each movie, we further
collected as many trailers as possible from YouTube, including
official and (serious) fan-based ones, or modern trailers for older
movies.  To evaluate the trailers produced by our algorithm, we also
collected a new held-out set of~41 movies. These movies were selected
from the Moviescope dataset\footnote{\href{http://www.cs.virginia.edu/~pc9za/research/moviescope.html}{http://www.cs.virginia.edu/~pc9za/research/moviescope.html}} \cite{cascante2019moviescope}, which
contains official movie trailers. The held-out set does not contain
any additional information, such as screenplays or TP annotations. The
statistics of TRIPOD$\bigoplus$ is presented in
Table~\ref{tab:dataset}.

\begin{table}[t]
\centering
\small
\begin{tabular}{@{}l@{~~}r@{}r@{~~}r@{}}
\hline
TRIPOD$\bigoplus$ & \multicolumn{1}{c}{Train} & \multicolumn{1}{c}{Dev+Test} & \multicolumn{1}{c}{Held-out}  \\ \hline
No. movies & 84 & 38 & 41 \\
No. scenes & 11,320 & 5,830 & --- \\
No. video shots & 81,400 & 34,100 & 48,600 \\
No. trailers & 277 & 155 & 41 \\% \hline
Avg. movie duration & 6.9k (0.6) & 6.9k (1.3) & 7.8k (2.3) \\
Avg. scenes per movie  & 133 (61) & 153 (54) & --- \\
Avg. shots per movie  & 968 (441) & 898 (339) & 1,186 (509) \\
\hfill duration & 6.9 (15.1) & 7.1 (13.6) & 6.6 (16.6) \\
Avg. valid shots per movie & 400 (131) & 375 (109) & 447 (111) \\
\hfill duration & 13.5 (19.0) & 13.9 (18.1) & 13.3 (19.6) \\
Avg. trailers per movie& 3.3 (1.0) & 4.1 (1.0) & 1.0 (0.0) \\
\hfill duration & 137 (42) & 168 (462) & 148 (20) \\
Avg. shots per trailer & 44 (27) & 43 (27) & 57 (28) \\
\hfill duration & 3.1 (5.8) & 3.9 (14.0) & 2.6 (4.0) \\
\hline
\end{tabular}
\vspace{-1ex}
\caption{Statistics of TRIPOD$\bigoplus$ dataset (for movies,
  screenplays, trailers);  standard deviation  within
  brackets, duration in seconds.}
  \vspace{-1em}
\label{tab:dataset}
\end{table}

\paragraph{Movie and Trailer Processing} The modeling approach put
forward in previous sections assumes that we know the correspondence
between screenplay scenes and movie shots. We obtain this mapping by
automatically aligning the dialogue in screenplays with subtitles
using Dynamic Time Warping (DTW;
\cite{myers1981comparative,papalampidi2020movie}).
We first segment the video into scenes based on this mapping, and then
segment each scene into shots using PySceneDetect\footnote{\href{https://github.com/Breakthrough/PySceneDetect}{https://github.com/Breakthrough/PySceneDetect}}. Shots
with less than 100 frames in total are too short for both processing
and displaying as part of the trailer and are therefore discarded.

Moreover, for each shot we extract visual and audio features. We
consider three different types of visual features: (1)~We sample one
key frame per shot and extract features using ResNeXt-101
\cite{xie2017aggregated} pre-trained for object recognition on ImageNet
\cite{deng2009imagenet}. (2)~We sample frames with a frequency of 1
out of every 10 frames (we increase this time interval for shots with
larger duration since we face memory issues) and extract motion
features using the two-stream I3D network pre-trained on Kinetics
\cite{carreira2017quo}. (3)~We use Faster-RCNN \cite{girshick2015fast}
implemented in Detectron2 \cite{wu2019detectron2} to detect person
instances in every key frame and keep the top four bounding boxes per
shot which have the highest confidence alongside with the respective
regional representations.
We first project all individual representations to the same lower
dimension and perform L2-normalization. Next, we consider the visual
shot representation as the sum of the individual vectors.
For the audio modality, we use YAMNet pre-trained on the
AudioSet-YouTube corpus \cite{gemmeke2017audio} for classifying audio
segments into 521 audio classes (e.g.,~tools, music, explosion); for
each audio segment contained in the scene, we extract features from
the penultimate layer. Finally, we extract textual features
\cite{papalampidi2020movie} from subtitles and screenplay scenes using
the Universal Sentence Encoder (USE; \cite{cer2018universal}).

For evaluation purposes, we need to know which shots in the movie are
trailer-worthy or not. We do this by segmenting the corresponding
trailer into shots and computing for each shot its visual similarity
with all shots in the movie. Shots with highest similarity values
receive positive labels (i.e.,~they should be in the trailer).
However, since trailers also contain shots that are not in the movie
(e.g.,~black screens with text, or simply material that did not make
it in the final movie), we also set a threshold below which we do not
map trailer shots to movie shots. In this way, we create
silver-standard binary labels for movie shots.

\vspace{-0.6em}
\paragraph{Sentiment Labels}
Since TRIPOD does not contain sentiment annotations, we instead obtain
silver-standard labels via COSMIC \cite{ghosal2020cosmic}, a
commonsense-guided framework with state-of-the-art performance for
sentiment and emotion classification in natural language
conversations. Specifically, we train COSMIC on MELD
\cite{poria2019meld}, which contains dialogues from episodes of the TV
series \textit{Friends} and is more suited to our domain than other
sentiment classification datasets
(e.g.,~\cite{busso2008iemocap,li2017dailydialog}).
After training, we use COSMIC to produce sentence-level sentiment
predictions for the TRIPOD screenplays. The sentiment of a scene
corresponds to the majority sentiment of its sentences.  We project
scene-based sentiment labels onto shots using the same one-to-many
mapping employed for TPs.

\section{Results and Analysis}
\label{sec:results}

\paragraph{Usefulness of Knowledge Distillation}
We first investigate whether we improve TP identification, as it is
critical to the trailer generation task. We split the set of movies
with ground-truth scene-level TP labels into development and test set
and select the top~5 (@5) and top~10 (@10) shots per TP in a movie. As
evaluation metric, we consider Partial Agreement (PA;
\cite{papalampidi2019movie}), which measures the percentage of TPs for
which a model correctly identifies \textit{at least} one ground-truth
shot from the 5 or 10 shots selected from the movie (see Appendix for
details).

\begin{table}[t]
\centering
\small
\begin{tabular}{l@{}r@{~~~}r}
\hline
 & PA@5$\uparrow$ & PA@10$\uparrow$ \\ \hline
 Random (evenly distributed) & 21.67 & 33.44 \\
 Theory position & 10.00 & 12.22 \\
 Distribution position & 12.22 & 15.56 \\
\textsc{Graph\-TP}\xspace~\cite{papalampidi2020movie} & 10.00 & 12.22 \\
\begin{tabular}[c]{@{}l@{}} Transformer \end{tabular} & 22.22 & 33.33 \\
\begin{tabular}[c]{@{}l@{}} \textsc{Graph\-Trailer}\xspace \end{tabular} & 27.78 & 35.56 \\
\hspace{0.1em}+ Screenplay, Asynchronous ($\mathcal{P}$)  & 28.89 & 41.11 \\
\hspace{0.1em}+ Screenplay, Asynchronous ($\mathcal{P}+\mathcal{R}$)  & 21.11 & 35.56 \\
\hspace{0.1em}+ Screenplay, Contrastive Joint ($\mathcal{P}+\mathcal{R}$) & 33.33 & 47.78 \\
\begin{tabular}[c]{@{}l@{}} \hspace{1em} + pre-training \end{tabular} & \textbf{34.44} & \textbf{50.00} \\
\hline
\end{tabular}
\vspace{-1ex}
\caption{Model performance on TP identification (test
  set). \textsc{Graph\-Trailer}\xspace shown with different training
  regimes. Evaluation metric: Partial Agreement (PA) against top~5
  (@5) and top~10 (@10) selected shots per TP and movie.}
\vspace{-1em}
\label{tab:TP_results}
\end{table}

Table~\ref{tab:TP_results} summarizes our results on the test set.  We
consider the following comparison systems: \textbf{Random} selects
shots from evenly distributed sections (average of 10 runs);
\textbf{Theory} assigns TP to shots according to screenwriting theory
(e.g.,~``Opportunity'' occurs at 10\% of the movie, ``Change of
plans'' at 25\%, etc.); \textbf{Distribution} selects shots based on
their expected position in the training data; \textbf{\textsc{Graph\-TP}\xspace} is the
original model of \cite{papalampidi2020movie} trained on screenplays
(we project scene-level TP predictions to shots); \textbf{Transformer}
is a base model without graph-related information.  We use our own
model, \textbf{\textsc{Graph\-Trailer}\xspace,} in several variants for TP
identification: without and with access to screenplays, trained only
with the prediction consistency loss ($\mathcal{P}$), both prediction
and representation losses ($\mathcal{P}+\mathcal{R}$), and our
contrastive joint training regime.

We observe that \textsc{Graph\-Trailer}\xspace outperforms all baselines, as well the
Transformer model. Although the latter encodes long-range dependencies
between shots, \textsc{Graph\-Trailer}\xspace additionally benefits from directly
encoding sparse connections learnt in the graph. Moreover,
asynchronous knowledge distillation via the prediction consistency
loss ($\mathcal{P}$) further improves performance, suggesting that
knowledge contained in screenplays is complementary to what can be
extracted from video.  Notice that when we add the representation
consistency loss ($\mathcal{P}+\mathcal{R}$), performance deteriorates
by a large margin, whereas the proposed training approach (contrastive
joint) performs best. Finally, pretraining offers further gains,
albeit small, which underlines the benefits of the screenplay-based
network.

\begin{table}[t]
\centering
\small
\begin{tabular}{L{16em}@{}rr}
\hline
 &  \multicolumn{1}{c}{Dev$\uparrow$} &  \multicolumn{1}{c}{Test$\uparrow$} \\ \hline
Random selection  w/o TPs &  14.47 & 5.61 \\
Random selection with TPs & 20.00 & 9.27 \\
\textsc{Text\-Rank}\xspace~\cite{mihalcea2004textrank} & 10.26 & 3.66 \\
\textsc{Graph\-Trailer}\xspace w/o TPs & 23.58 & 11.53 \\
\textsc{Graph\-Trailer}\xspace with TPs & \textbf{26.95} & \textbf{16.44} \\ \hline
CCANet~\cite{wang2020learning} & 31.05 & 15.12 \\
Transformer & 32.63 & 17.32 \\
Supervised \textsc{Graph\-Trailer}\xspace & \textbf{33.42} & \textbf{17.80} \\
\hline
Upper bound & 86.41 & \multicolumn{1}{c}{---} \\ \hline
\end{tabular}
\vspace{-1ex}
\caption{Performance of unsupervised (upper part) and weakly
  supervised (lower part) models on trailer generation: accuracy of
  correctly identified trailer shots. All systems have the same shot
  budget for trailer creation.}
\vspace{-1em}
\label{tab:trailer_results}
\end{table}

\paragraph{Trailer Quality} We now evaluate the trailer
generation algorithm of \textsc{Graph\-Trailer}\xspace on the held-out set of 41 movies
(see Table~\ref{tab:dataset}).  As evaluation metric, we use accuracy,
i.e.,~the percentage of correctly identified trailer shots and we
consider a total budget of 10 shots for the trailers in order to
achieve the desired length ($\sim$2~minutes).

We compare \textsc{Graph\-Trailer}\xspace against several unsupervised approaches
(first block in Table~\ref{tab:trailer_results}) including:
\textbf{Random selection} among all shots and among TPs identified by
\textsc{Graph\-Trailer}\xspace; we also implement two graph-based systems based on a
fully-connected graph, where nodes are shots and edges denote the
degree of similarity between them. This graph has no knowledge of TPs,
it is constructed by calculating the similarity between generic
multimodal
representations. \textbf{\textsc{Text\-Rank}\xspace}~\cite{mihalcea2004textrank}
operates over this graph to select shots based on their centrality,
while \textbf{\textsc{Graph\-Trailer}\xspace without TPs} traverses the graph with TP
and sentiment criteria removed (Equation~\ref{eq:score}). For the unsupervised systems which include stochasticity and produce proposals (Random, \textsc{Graph\-Trailer}\xspace), we consider the best proposal trailer. The second
block of Table~\ref{tab:trailer_results} presents supervised
approaches which use noisy trailer labels for training. These include
\textbf{CCANet} \cite{wang2020learning}, which only considers visual
information and computes the cross-attention between movie and trailer
shots, and a vanilla \textbf{Transformer} trained for the binary task
of identifying whether a shot should be in the trailer without
considering screenplays, sentiment or TPs.
  \textbf{Supervised
  \textsc{Graph\-Trailer}\xspace} consists of our video-based network trained on the
same data as the Transformer.

\textsc{Graph\-Trailer}\xspace performs best among unsupervised methods. Interestingly,
\textsc{Text\-Rank}\xspace is worse than random, illustrating that tasks like trailer
generation cannot be viewed as standard summarization
problems. \textsc{Graph\-Trailer}\xspace without TPs still performs better than
\textsc{Text\-Rank}\xspace and random TP selection.\footnote{Performance on the test
  set is lower because we only consider trailer labels from the
  official trailer, while the dev set contains multiple trailers.}
With regard to supervised approaches, we find that using all
modalities with a standard architecture (Transformer) leads to better
performance than sophisticated models using visual similarity
(CCANet). By adding graph-related information (Supervised
\textsc{Graph\-Trailer}\xspace), we obtain further improvements.

\begin{table}[t]
\centering
\small
\begin{tabular}{lr}
\hline
 &  \textbf{Accuracy} $\uparrow$ \\ \hline
\textsc{Graph\-Trailer}\xspace & 22.63 \\
\hspace{0.1em}+ Auxiliary, Asynchronous ($\mathcal{P}$) & 21.87 \\
\hspace{0.1em}+ Auxiliary, Asynchronous ($\mathcal{P}+\mathcal{R}$) & 22.11 \\
\hspace{0.1em}+ Auxiliary, Contrast Joint ($\mathcal{P}+\mathcal{R}$) & 25.44 \\
 \hspace{1em} + pre-training & \textbf{25.79} \\
\hline
\end{tabular}
\caption{Different training regimes for \textsc{Graph\-Trailer}\xspace: accuracy (\%) of correctly identified trailer
  shots. For direct comparison \textsc{Graph\-Trailer}\xspace here only uses the
  narrative structure criterion, no information about sentiment
  intensity is considered.}
\vspace{-1em}
\label{tab:ablation_training}
\end{table}

\begin{table}[t]
\centering
\small
\begin{tabular}{L{16em}r}
\hline
 &  \textbf{Accuracy} $\uparrow$ \\
 \hline
Similarity & 24.28 \\
Similarity + TPs & 25.79 \\
Similarity + sentiment & 23.97 \\
Similarity + TPs + sentiment & \textbf{26.95} \\
\hline
\end{tabular}
\caption{\textsc{Graph\-Trailer}\xspace with different
  criteria for performing random walks in the movie graph (Algorithm 1,
  Equation~\eqref{eq:score}).}
\vspace{-1em}
\label{tab:ablation_criteria}
\end{table}

We perform two ablation studies on the development set for
\textsc{Graph\-Trailer}\xspace. The first study aims to assess how the different
training regimes of the dual network influence downstream trailer
generation performance. We observe in
Table~\ref{tab:ablation_training} that asynchronous training does not
offer any discernible improvement over the base model. However, when
we jointly train the two networks (video- and screenplay-based) using
prediction and representation consistency losses, performance
increases by nearly~3\%. A further small increase is observed when the
screenplay-based network is pre-trained on more data.

The second ablation study concerns the criteria used for performing
random walks on the graph $\mathcal{G}$. As shown in Table~\ref{tab:ablation_criteria}, when
we enforce the nodes in the selected path to be close to key events
(similarity + TPs) performance improves. When we rely solely on
sentiment (similarity + sentiment), performance drops slightly. This
suggests that in contrast to previous approaches
which mostly focus on superficial visual attractiveness
\cite{xu2015trailer,wang2020learning} or audiovisual sentiment
analysis \cite{smith2017harnessing}, sentiment information on its own
is not sufficient and may promote outliers that do not fit well in a
trailer. On the other hand, when sentiment information is combined
with knowledge about narrative structure (similarity + TPs +
sentiment), we observe the highest accuracy. This further validates
our hypothesis that the two theories about creating trailers
(i.e.,~based on narrative structure and emotions) are complementary and
can be combined.

Finally, since we have multiple trailers per movie (for the dev set), we
can measure the overlap between their shots (\textbf{Upper
  bound}). The average overlap is 86.14\%, demonstrating good
agreement between trailer makers and a big gap between human
performance and automatic models.

\begin{table}[t]
\centering
\small
\begin{tabular}{@{}l@{~~}r@{~~~}r@{~~~}r@{~~~}r@{~~~}r@{}}
\hline
 & \multicolumn{1}{c}{Q1} & \multicolumn{1}{c}{Q2} & Best & Worst & BWS \\
 \hline
Random selection w/o TPs & 38.2 & 45.6 & 19.1 & 25.9 & -1.26 \\
\textsc{Graph\-Trailer}\xspace w/o TPs & 37.2 & 44.5 & \textbf{24.4} & 25.9 & -0.84 \\
\textsc{Graph\-Trailer}\xspace w/ TPs & \textbf{41.4} & \textbf{48.2} & 20.8 & \textbf{11.6} & \textbf{1.40} \\
\hline
CCANet & 37.7 & 46.6 & 14.3 & 15.2 & -0.14  \\
Superv. \textsc{Graph\-Trailer}\xspace & 37.7 & 47.1 & 21.4 & 21.4 & 0.84 \\
\hline
\end{tabular}
\vspace{-1ex}
\caption{Human evaluation on held-out set. Percentage of Yes
  answers for: Does the trailer contain sufficient information (Q1)
  and is it attractive (Q2). Percentage of times each system was
  selected as Best or Worst, and standardized best-worst scaling
  score.}
  \vspace{-1em}
\label{tab:human_evaluation}
\end{table}

\paragraph{Human Evaluation}
We also conducted a human evaluation study to assess the quality of
the generated trailers. For human evaluation, we include Random
selection without TPs as a lower bound, the two best performing
unsupervised models (i.e.,~\textsc{Graph\-Trailer}\xspace with and without TPs), and
two supervised models: CCANet, which is the previous state of the art
for trailer generation, and the supervised version of our model, which
is the best performing model according to automatic
metrics.\footnote{We do not include ground-truth trailers in the
  human evaluation, since they are post-processed (i.e.,~montage,
  voice-over, music) and thus not directly comparable to automatic
  ones.}  We generated trailers for all movies in the held-out set. We
then asked Amazon Mechanical Turk (AMT) crowd workers to watch all
trailers for a movie, answer questions relating to the information
provided (Q1) and the attractiveness (Q2) of the trailer, and select
the best and worst trailer. We collected assessments from five
different judges per movie.

Table~\ref{tab:human_evaluation} shows that \textsc{Graph\-Trailer}\xspace with TPs
provides on average more informative (Q1) and attractive (Q2) trailers
than all other systems. Although \textsc{Graph\-Trailer}\xspace without TPs and
Supervised \textsc{Graph\-Trailer}\xspace are more often selected as best, they are
also chosen equally often as worst. When we compute standardized
scores (z-scores) using best-worst scaling \cite{bestworst}, \textsc{Graph\-Trailer}\xspace with TPs
achieves the best performance (note that is also rarely selected as
worst) followed by Supervised \textsc{Graph\-Trailer}\xspace. Interestingly,
\textsc{Graph\-Trailer}\xspace without TPs is most often selected as best (24.40\%),
which suggests that the overall approach of modeling movies as graphs
and performing random walks instead of individually selecting shots
helps create coherent trailers. However, the same model is also most
often selected as worst, which shows that this naive approach on its
own cannot guarantee good-quality trailers.

We include video examples of generated trailers based on our approach
in the Supplementary Material. Moreover, we provide a step-by-step graphical
example of our graph traversal algorithm in the Appendix.

\paragraph{Spoiler Alert!} Our model does not \emph{explicitly}
avoid spoilers in the generated trailers.  We experimented with a
spoiler-related criterion when traversing the movie graph in
Algorithm~\ref{algorithm}. Specifically, we added a penalty when
selecting shots that are in ``spoiler-sensitive'' graph
neighborhoods. We identified such neighborhoods by measuring the
shortest path from the last two TPs, which are by definition the
biggest spoilers in a movie. However, this variant of our algorithm
resulted in inferior performance and we thus did not pursue it
further. We believe that such a criterion is not beneficial for
proposing trailer sequences, since it discourages the model from
selecting exciting shots from the latest parts of the movie. These
high-tension shots are important for creating interesting trailers and
are indeed included in real-life trailers. More than a third of
professional trailers in our dataset contain shots from the last two
TPs (``Major setback'', ``Climax''). We discuss this further in
the Appendix.

We also manually inspected the generated trailers and
found that spoilers are not very common (i.e.,~we identified one major spoiler shot in a random sample of 12 trailers from the test set), possibly because the
probability of selecting a major spoiler is generally low.  And even
if a spoiler-sensitive shot is included, when taken out of context
it might not be enough to unveil the ending of a movie. However, we
leave it to future work to investigate more elaborate spoiler
identification techniques, which can easily be integrated to our
algorithm as extra criteria.

\section{Conclusions}

In this work, we proposed a trailer generation approach which adopts a
graph-based representation of movies and uses interpretable criteria
for selecting shots.  We also show how privileged information from
screenplays can be leveraged via contrastive learning, resulting in a
model that can be used for turning point identification \emph{and} trailer
generation. Trailers generated by our model were judged favorably
in terms of their content and attractiveness.

In the future we would like to focus on methods for predicting
fine-grained emotions (e.g.,~grief, loathing, terror, joy) in movies.
In this work, we consider positive/negative sentiment as a stand-in
for emotions, due to the absence of in-domain labeled
datasets. Previous efforts have focused on tweets
\cite{abdul-mageed-ungar-2017-emonet}, Youtube opinion videos
\cite{bagher-zadeh-etal-2018-multimodal}, talkshows
\cite{Grimm:ea:2008}, and recordings of human interactions
\cite{BILAKHIA201552}.  Preliminary experiments revealed that
transferring fine-grained emotion knowledge from other domains to ours
leads to unreliable predictions compared to sentiment which is more
stable and improves trailer generation performance. Avenues for future
work include new emotion datasets for movies, as well as emotion
detection models based on textual \emph{and} audiovisual cues.

{

    \small
    \bibliographystyle{ieee_fullname}

\section{Model Details}

In this section we provide details about the various modeling
components of our approach. We begin by providing details of the \textsc{Graph\-Trailer}\xspace architecture
(Section~\ref{sec:textbfvideograph}), then move to discuss how the TP
identification network is trained (Section~\ref{sec:train-tp-ident}),
and finally give technical details about pre-training on screenplays
(\ref{sec:pre_training_appendix}), and the sentiment flow used for graph traversal
(\ref{sec:sent-pred-desir}).

\subsection{\textbf{\textsc{Graph\-Trailer}\xspace}}
\label{sec:textbfvideograph}

As mentioned in the main paper, we assume a multimodal
representation~$h_i$ per shot based on movie subtitles, audio, and
video frames. We then construct matrix~$E$ by computing the pairwise
similarity between all pairs of shots $h_i,h_j$ in a movie:
\begin{gather}
     e_{ij} = \tanh(W_{i}h_{i} + b_{i})
            \tanh(W_{j}h_{j}+ b_{j}) + b_{ij}
\end{gather}
Similarities are normalized using the softmax function (row-wise
normalization in matrix $E$). We thus obtain a fully-connected
directed graph where edge $e'_{ij}$ records the probability that~$h_i$
is connected with $h_j$ in the graph. Next, we remove all edges from
the graph which represent connections from future to past shots
(i.e.,~we render adjacency matrix~$E'$ upper triangular by masking the
positions below the diagonal). Finally, we sparsify the graph by
selecting a top-$k$ neighborhood $\mathcal{P}_i$ per shot $h_i$:
$\mathcal{P}_i = \operatorname{argmax}\limits_{j\in[1,M],
  |\mathcal{P}_i|=k}e'_{ij}$. Instead of deciding on a fixed number of
$k$~neighbors for all shots and movies, we determine a predefined set
of options for~$k$ (e.g., integer numbers contained in a set $O$) and
learn to select the appropriate $k$ per shot via a parametrized
function: $z_{i} = W_{n}e'_{i} + b_{n}$, where $z_i$ is a probability
distribution over the neighborhood size options for shot~$h_i$. Hence,
the final neighborhood size for shot~$h_i$ is:
$k_i=\operatorname{argmax}_{t\in O}z_{it}$.

We address discontinuities in our model (i.e., top-$k$ sampling,
neighborhood size selection) by utilising the Straight-Through
Estimator \cite{bengio2013estimating}. During the backward pass we
compute the gradients with the Gumbel-softmax reparametrization trick
\cite{maddison2017concrete,jang2017categorical}.  The same procedure
is followed for constructing and sparsifying scene-level graphs in the
auxiliary screenplay-based network.

\subsection{Training on TP Identification}
\label{sec:train-tp-ident}

Section~\ref{sec:problem_formulation} presents our training regime for the video- and screenplay-based model assuming TP labels
for scenes are available (i.e., binary labels indicating whether a
scene acts as a TP in a movie). Given such labels, our model is
trained with a binary cross-entropy loss (BCE) objective between the
few-hot gold labels and the network's TP predictions.

However, in practice, our training set contains silver-standard labels
for scenes. The latter are released together with the TRIPOD
\cite{papalampidi2019movie} dataset and were created automatically.
Specifically, TRIPOD provides \emph{gold-standard} TP annotations for
synopses (not screenplays), under the assumption that synopsis
sentences are representative of TPs. And sentence-level annotations
are projected to scenes with a matching model trained with teacher
forcing \cite{papalampidi2019movie} to create silver-standard labels.

In this work, we consider the probability distribution over screenplay
scenes as produced by the teacher model for each TP and compute the KL
divergence loss between the teacher posterior distributions $q(y_t)$
and the ones computed by our model~$p(y_t)$:
\begin{gather}
 \mathcal{O}_t = \mathcal{D}_{KL}\left(p(y_t) \middle\| q(y_t) \right), t \in [1,T] \label{eq:KL_TPs}
\end{gather}
where $T$ is the number of TPs. Hence, the loss $\mathcal{S}$
(Equation~\eqref{eq:loss}, Section~\ref{sec:kd}) used for the auxiliary screenplay network
is: $\mathcal{S}=\sum_{t=1}^T\mathcal{O}_t$ . Accordingly, for the video-based network which operates over shots, we first aggregate  shot
probabilities to scene-level ones (see Sections~3.3 and~4) and then
compute the KL divergence between the aggregated probabilities and the
teacher distribution (loss $\mathcal{V}$ in Equation~\eqref{eq:loss},
Section~\ref{sec:kd}), in the same fashion as in Equation \eqref{eq:KL_TPs}.

\subsection{Self-supervised Pre-training}
\label{sec:pre_training_appendix}

In Section~\ref{sec:pretraining}, we describe a modified approach to Contrastive
Predictive Coding (CPC; \cite{oord2018representation}) for
pre-training the screenplay-based network on a larger corpus of screenplays
(i.e.,~Scriptbase \cite{gorinski-lapata-2015-movie}). The training
objective is InfoNCE (see Equation~\eqref{eq:info_nce}) and
considers structure-aware representations for each scene~$s_i$ in the
screenplay.

Here, we explain in more detail how these representations are
computed. Given scene~$s_i$, we select the next node in the path based
on the edge weights connecting it to its neighbors
$s_t=\operatorname{argmax}\limits_{j\in \mathcal{P}_i}e'_{ij}$ where
$\mathcal{P}_i$ is the sparsified immediate neighborhood for $s_i$. We
perform $k$ such steps and create path $p_i$ which contains
(graph-based) representations (via one-layer GCN) for the selected
nodes. Finally, representation $g_i$ for scene $s_i$ is computed from
this path $p_i$ via mean-pooling.

Notice that we compute structure-aware scene representations~$g_i$ via
random walks rather than stacking multiple GCN layers.  We could in
theory stack two or three GCN layers and thus consider many more
representations (e.g.,~100 or 1,000) for contextualizing~$g_i$.
However, this would result to over-smoothed representations, that
converge to the same vector \cite{oono2019graph}. Moreover, when
trying to contextualize such large neighborhoods in a graph, the
bottleneck phenomenon prevents the effective propagation of long-range
information \cite{alon2020bottleneck} which is our main goal. Finally,
in ''small world'' networks with a few hops, neighborhoods could end
up containing the majority of the nodes in the graph
\cite{barcelo2019logical}, which again hinders meaningful exploration
of long-range dependencies. Based on these limitations of GCNs, we
perform random walks, which allow us to consider a small number
of representations when contextualizing a scene, while also exploring
long-range dependencies.

\subsection{Sentiment Flow in \textsc{Graph\-Trailer}\xspace}
\label{sec:sent-pred-desir}

One of the criteria for
selecting the next shot in our graph traversal algorithm (Section~\ref{sec:graph_traversal})
is the sentiment flow of the trailer generated so far. Specifically,
we adopt the
hypothesis\footnote{\tiny{\url{https://www.derek-lieu.com/blog/2017/9/10/the-matrix-is-a-trailer-editors-dream}}}
that trailers are segmented into three sections based on sentiment
intensity. The first section has medium intensity for attracting
viewers, the second section has low intensity for delivering key
information about the movie and finally the third section displays
progressively higher intensity for creating cliffhangers and excitement
for the movie.

Accordingly, given a budget of $L$~trailer shots, we expect the
first~$L/3$ ones to have medium intensity without large variations
within the section (e.g., we want shots with average absolute
intensity close to 0.7, where all scores are normalized to a range
from -1 to 1). In the second part of the trailer (i.e.,~the next $L/3$
shots) we expect a sharp drop in intensity and shots within this
section to maintain more or less neutral sentiment (i.e.,~0
intensity). Finally, for the third section (i.e.,~the final $L/3$
shots) we expect intensity to steadily increase. In practice, we
expect the intensity of the first shot to be~0.7 (i.e.,~medium
intensity), increasing by 0.1 with each subsequent shot until
we reach a peak at the final shot.

\section{Implementation Details}

\paragraph{Evaluation Metrics} Previous work
\cite{papalampidi2019movie} evaluates the performance of TP
identification models in terms of three metrics: Total Agreement (TA),
i.e., the percentage of TP scenes that are correctly identified,
Partial Agreement (PA), i.e., the percentage of TP events for which at
least one gold-standard scene is identified, and Distance (D), i.e.,
the minimum distance in number of scenes between the predicted and
gold-standard set of scenes for a given TP, normalized by the
screenplay length. We report results with the partial agreement
metric. We can no longer use total agreement, since we evaluate
against silver standard (rather than gold) labels for shots (rather
than scenes) and as a result consider all shots within a scene equally
important.  We do not use the distance metric either since it yields
very similar outcomes and does not help discriminate among model
variants.

\paragraph{Hyperparameters} Following previous work
\cite{papalampidi2020movie}, we project all types of features
(i.e.,~textual, visual, and audio) to the same lower dimension
of~128. We find that larger dimensions increase the number of
parameters considerably and yield inferior results possibly due to
small dataset size.

We contextualize scenes (with respect to the screenplay) and shots
(with respect to the video) using transformer encoders. We
experimented with 2, 3, 4, 5, and 6 layers in the encoder and obtained
best results with 3~layers. For the feed forward (FF) dimension, we
experimented with both a standard size of~2,048 and a smaller size
of~1,024 and found the former works better. We use another transformer
encoder to compute the representation of a scene from a sequence of
input \emph{sentence} representations. This encoder has 4 layers and
1,024 FF dimension.  Both encoders, employ~8 attention heads and
0.3~dropout.

During graph sparsification (i.e.,~selection of top-$k$ neighbors), we
consider different neighborhood options for the scene- and shot-based
networks due to their different granularity and size. Following
\cite{papalampidi2020movie}, we consider \mbox{[1--6]}~neighbors for
the scene network and we increase the neighborhood size to [6--12] for
the shot network.

For training our dual network on TP identification, we use the
$\mathcal{L}_{TP}$ objective described in Equation~\eqref{eq:loss}
(Section~\ref{sec:kd}). We set hyperparameters~$a$ and~$b$ that determine the
importance of the prediction and representation consistency losses in
$\mathcal{L}_{TP}$ to 10 and 0.03, respectively.  Moreover, while
pre-training the screenplay-based network on the Scriptbase corpus
\cite{gorinski-lapata-2015-movie} we select as future context window
10\% of the screenplay. As explained in Sections~\ref{sec:pretraining}
and~\ref{sec:pre_training_appendix}, we
compute structure-aware scene representations by performing random
walks of~$k$ steps in the graph starting from an anchor scene. We
empirically choose 3~steps.

Finally, as described in Section~\ref{sec:graph_traversal}, \textsc{Graph\-Trailer}\xspace uses a combination
of multiple criteria for selecting the next node to be included in the
trailer path. The criteria are combined using hyperparameters which
are tuned on the development set. The search space for each
hyperparameter~$\lambda$ (Equation~\eqref{eq:score}, Section~\ref{sec:graph_traversal}) is [0, 1, 5, 10,
  15, 20, 25, 30] and we find that the best combination for
$\lambda_1$ (semantic similarity), $\lambda_2$ (time proximity),
$\lambda_3$ (narrative structure), and $\lambda_4$ (sentiment intensity)
is [1, 5, 10, 10], respectively.

\section{Results: Ablation Studies}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{per_section_trailer_shots.png}
    \caption{Distribution of trailer shots corresponding to different
      sections of a movie (development set) as determined by TPs.
      Trailer shots come from all parts of the movie, even from the
      end, although the majority are from the beginning and
      middle.}
    \label{fig:tps_distribution}
\end{figure}

\begin{table}[t]
\centering
\small
\begin{tabular}{L{18em}r}
\hline
Opportunity & 52.63 \\
Change of plans & 55.26 \\
Point of no return & 47.37 \\
Major setback & 34.21 \\
Climax & 34.21 \\
\hline
\end{tabular}
\caption{Percentage (\%) of trailers that include at least one shot labeled as a specific type of TP on the development set. The first two TPs (that present an introduction to the story) appear more frequently in trailers, especially in comparison with the last two, which often contain major spoilers.}
\label{tab:trailers_vs_trailers}
\end{table}

\begin{table}[t]
\centering
\small
\begin{tabular}{L{12em}r}
\hline
 &  \textbf{Sentiment intensity} \\
 \hline
First part & 11.50 \\
Second part & 9.35 \\
Third part & 14.75 \\
\hline
\end{tabular}
\caption{Average absolute sentiment intensity per trailer section,
  when we divide the trailers into three even parts (development
  set).}
  \vspace{-1em}
\label{tab:sentiment_per_section}
\end{table}

\begin{figure*}[h!]
    \centering
\begin{tabular}{|c|c|} \hline
        {{\small Step 1 - Initialize path: sample shot from TP1 predictions}} & \begin{tabular}[c]{c} {{\small Step 2: Select next shot from immediate neighborhood}} \end{tabular}  \\
         \includegraphics[width=0.46\textwidth]{example_graphtrailer.pdf} &
         \includegraphics[width=0.46\textwidth]{example_graph_step2.pdf} \\ \hline

        {{\small Step 3: Select next shot from immediate neighborhood}} & {{\small Step 4: Select next shot  from immediate neighborhood}}   \\ & \\
        \includegraphics[width=0.46\textwidth]{example_graph_step3.pdf} &
        \includegraphics[width=0.46\textwidth]{example_graph_step4.pdf} \\\hline

\end{tabular}
 \caption{Run of \textsc{Graph\-Trailer}\xspace algorithm for the movie ''The
   Shining''. Step 1 illustrates the shot-level graph (pruned
   for better visualization) with colored nodes representing the
   different types of TPs predicted in the movie (i.e., { \color{tp1}
     TP1}, { \color{tp2} TP2}, { \color{tp3} TP3}, { \color{tp4} TP4},
   { \color{tp5} TP5}).  Our algorithm starts by sampling a shot
   identified as { \color{tp1} TP1} by \textsc{VideoGraph}\xspace (Step 1). For each
   next step, we only consider the immediate neighborhood of the
   current shot (i.e.,~6--12 neighbors) and select the next shot based
   on the following criteria: (1) semantic similarity, (2) time
   proximity, (3) narrative structure, and (4) sentiment intensity
   (Steps 2-4). Our algorithm continues in
   Figure~\ref{fig:random_walk_2}.
   \label{fig:random_walk_1}}
\end{figure*}

\begin{figure*}[h!]
    \centering
\begin{tabular}{|c|c|} \hline
        {{\small Step 5: Select next shot  from immediate neighborhood}} & {{\small Step 6: Select next shot  from immediate neighborhood}}   \\ & \\
        \includegraphics[width=0.46\textwidth]{example_graph_step6.pdf} &
        \includegraphics[width=0.46\textwidth]{example_graph_step7.pdf} \\\hline
        {{\small Step 7: Select next shot  from immediate neighborhood}} & \begin{tabular}[c]{c} {{\small Step 8: Create trailer based on retrieved sequence of shots}} \end{tabular}  \\ & \\
         \includegraphics[width=0.46\textwidth]{example_graph_step8-compressed.pdf} &
         \includegraphics[width=0.46\textwidth]{example_graph_complete_path.pdf} \\ \hline
\end{tabular}
 \caption{We continue to build the trailer path by selecting the next
  shot from the immediate neighborhood based on interpretable
  criteria (Steps 5--7). Finally, we assemble the proposal trailer by
  concatenating the shots in the path. Our algorithm allows users to
  review candidate shots at each step and manually select the best
  one while taking into account our criteria.  \textsc{Graph\-Trailer}\xspace allows
  users to create trailers by only reviewing around 10\% of the
  movie based on recommendations which are interpretable
  (e.g.,~coherence with previous shot, relevance to story or
  intensity).
  \label{fig:random_walk_2}}
\end{figure*}

\section{Task Decomposition Analysis}

\paragraph{How Narrative Structure Connects with Trailers} According
to screenwriting theory \cite{Hauge:2017}, the five TPs segment movies
into six thematic units, namely, ''Setup'', ''New Situation'',
''Progress'', ''Complications and Higher Stakes'', ''Final Push'', and
''Aftermath''. To examine which parts of the movie are most prevalent
in a trailer, we compute the distribution of shots per thematic unit
in \emph{gold} trailers (using the extended development set of
TRIPOD). As shown in Figure~\ref{fig:tps_distribution}, trailers on
average contain shots from all sections of a movie, even from the last
two, which might reveal the ending. Moreover, most trailer shots
(30.33\%) are selected from the middle of the movie (i.e.,~Progress)
as well as from the beginning (i.e.,~16.62\% and 25.45\% for ``Setup''
and ``New Situation'', respectively). These empirical observations
corroborate industry principles for trailer
creation.\footnote{\tiny{\url{https://archive.nytimes.com/www.nytimes.com/interactive/2013/02/19/movies/awardsseason/oscar-trailers.html?_r=0}}}

Next, we find how often the trailers include the different types of
key events denoted by TPs. We present the percentage of trailers (on
the development set) that include at least one shot per TP in
Table~\ref{tab:trailers_vs_trailers}. As can be seen, more than half
of the trailers (i.e., 52.63\% and 55.26\%) include shots related to
the first two TPs, whereas only 34.21\% of trailers have any
information about the two final ones. This is expected, since the
first TPs are introductory to the story and hence  more important
for making trailers, whereas the last two may contain spoilers and are
often avoided.

\paragraph{How Sentiment Connects with Trailers} Empirical rules for
trailer
making\footnote{\tiny{\url{https://www.derek-lieu.com/blog/2017/9/10/the-matrix-is-a-trailer-editors-dream}}}
suggest that a trailer should start with shots of medium intensity to
captivate the viewers, then decrease the sentiment intensity in order
to deliver key information about the movie, and finally build up the
tension until it reaches a climax.

Here, we analyze the sentiment flow in real trailers from our
development set based on predicted sentiment scores (see Sections~\ref{sec:sentiment_prediction}
and 4). Specifically, we compute the absolute sentiment intensity
(i.e.,~regardless of positive/negative polarity) per shot in the
(true) trailers.  In accordance with our experimental setup, we again
map trailer shots to movie shots based on visual similarity and
consider the corresponding sentiment scores predicted by our
network. We then segment the trailer into three equal sections and
compute the average absolute sentiment intensity per section. In
Table~\ref{tab:sentiment_per_section} presents the results. As
expected, on average, the second part is the least intense, whereas
the third has the highest sentiment intensity. Finally, when we again
segment each trailer into three equal sections and measure the
sentiment flow from one section to the next, we find that 46.67\% of
the trailers follow a ''V'' shape, similar to our sentiment condition
for generating proposal trailers with \textsc{Graph\-Trailer}\xspace.

\paragraph{Examples of Walks in \textsc{Graph\-Trailer}\xspace} We present in
Figures~\ref{fig:random_walk_1} and~\ref{fig:random_walk_2} a real
example of how \textsc{Graph\-Trailer}\xspace operates over a sparse (shot) graph for
the movie ''The Shining''. Here, we show the algorithm's inner
workings on a further pruned graph for better visualization (Step 1;
Figure~\ref{fig:random_walk_1}), while in reality we use the full
graph as input to \textsc{Graph\-Trailer}\xspace.

We begin with shots that have been identified as TP1
(i.e.,~''Opportunity''; introductory event for the story). We sample a
shot (i.e., bright green nodes in graph) and initialize our path. For
the next steps (2--7; in reality, we execute up to 10~steps, but we
excluded a few for brevity), we only examine the immediate
neighborhood of the current node and select the next shot to be
included in the path based on the following criteria: (1) semantic
coherence, (2) time proximity, (3) narrative structure, and (4)
sentiment intensity. We give more details about how we formalize and
combine these criteria in Section~\ref{sec:graph_traversal}.

We observe that our algorithm manages to stay close to important
events (colored nodes) while creating the path, which means that we
reduce the probability of selecting random shots that are irrelevant
to the main story. Finally, in Step 8, Figure~\ref{fig:random_walk_2},
we assemble the proposal trailer by concatenating all shots in the
retrieved path. We also illustrate the path in the graph (i.e.,~red
line).

An advantage of our approach is that it is interpretable and can be
easily used as a tool with a human in the loop. Specifically, given
the immediate neighborhood at each step, one could select shots based
on different automatic criteria or even manually. Our approach
drastically reduces the amount of shots that need to be reviewed to
create trailer sequences to 10\% of the movie.  Moreover, our criteria
allow users to explore different sections of the movie, and create
diverse trailers.

redpajama_set_name: RedPajamaArXiv


According to the Global Air Report 2019, toxic air will reduce the life expectancy of children exposed to it by 20 months. Air pollution is a serious problem around the world and toxic air holds the infamous title of the fifth leading factor in mortality across the world.
One of the most deadliest diseases for amphibians, chytridiomycosis, is responsible for eliminating about 30 species in the early 2000s. With the added problem of habitat destruction due to climate change, many amphibians are in real danger.


I have spoken at conferences to encourage application developers to implement two-factor authentication (2FA), also known as multi-factor authentication (MFA), in order to address the inherent problems with username+password authentication systems. Enabling users to set up multi-factor authentication is one of the best ways to improve application security and protect user data.
Users should be encouraged, perhaps even compelled, to choose more than one of these options. Otherwise, there arises a significant risk of users getting themselves locked out of their accounts if they, for example, lose their U2F keys or TOTP code generators. While such predicaments could theoretically be addressed by out-of-band customer service requests, that requires customer service staff to reliably and correctly authenticate users via telephone calls and other inherently insecure mechanisms. No matter how you look at it, account lock-outs yield poor user experiences, customer service hassles, and security risks, and therefore account lockouts should be avoided at all costs.
So if the user selects U2F, the system should encourage the user to also set up TOTP and/or single-use recovery codes. One might go so far as to not even activate multi-factor authentication until at least two methods are set up.
TOTP is a standardized protocol and thus should be offered in such a way that it works with commonly-used mobile apps such as 1Password, Authenticator, and Authy. TOTP is vulnerable to phishing attacks and thus not as secure as U2F keys, but unlike U2F keys it can be readily used on mobile devices, which is an argument for encouraging users to set it up even if they normally use U2F keys for everyday access.
Single-use emergency codes are not very useful on their own, but they are straightforward and pair well with both U2F and TOTP. The primary challenge is ensuring that users store these recovery codes in a password manager or other secure location.
While multi-factor authentication presents challenges for both application development and end-user experiences, it is increasingly important for developers to offer it as an option to users who have the requisite knowledge and motivation to utilize it.
Have I missed any important best practices? Other comments you'd like to share? Reach out and let me know.


text:
Now is the Time to Embrace Apartment Sustainability and Energy Savings
RealPage Blog
CATEGORY Property Operations
Would you like to get the latest insights on real estate market trends and technology?
By Tim Blackwell
When it comes to utility management and apartment sustainability, multifamily housing can shape its future by becoming more agile and adaptable, and embracing change. And there is no better time than now, industry leaders say, to realize opportunities in conservation that trickle to the bottom line through operational efficiencies.
The proliferation of the Internet of Things has put technology in place that allows operators and residents to save on utility costs. Investors are increasingly emphasizing sustainable apartments, and residents from Millennials to Baby Boomers want more eco-friendly places to live. Talk of reining in utility expenses has become common at industry round tables.
Jeff Kok, Chief Innovation Officer and Chief Information Officer at Mill Creek Residential Trust, says it's a very interesting time for sustainability and associated environmental, social, and governance criteria. He was among a number of sustainability experts in multifamily who spoke at February's 2021 RealPage Energy Summit.
"You are seeing a much stronger push with investors across the board," Kok said at the Summit. "They are looking for programs that tie to sustainability and ESG. The energy performance of every asset is critical, especially when rents are down in markets like San Francisco and owner/operators are looking to reduce risk, enhance returns, and drive superior quality and performance."
Adding asset value by reducing expenses
The success of a sustainable program depends on buy-in from stakeholders and residents, multifamily industry leaders agreed.
Providing residents with the tools to promote sustainability – including education and consumption metrics through monthly billings – is a starting point. If residents can see their consumption and its impact on billing, they are more likely to try to conserve.
Vicki Parrish, Sr. Director of Advantage Solutions at Greystar, says her company has gone to great lengths to educate residents on sustainability and how it benefits those who live in a community with an eco-friendly mission.
Sustainability adds value to an asset, and it's not hard to achieve. Programs that reimburse for low-flow toilets and faucets, as well as solar programs where expenses are abated or credited back, are widely available. Their financial impact makes properties more attractive to residents and equally alluring to investors.
"There are so many ways to reduce expenses though sustainability," Parrish said. "We find that the more the building has sustainable features and programs, the more desirable the community is and occupancy rates for ownership go up."
Trashing paper billing methods in favor of electronic billing is a simple, effective way to start a sustainable footprint. Last year, Greystar began e-billing at most of the 700,000 units the company manages while enabling a more interactive experience with residents.
Greystar educates tenants about sustainable practices through company blogs and provides usage data so they understand what they are consuming.
"We try very hard to educate residents, among other things communicating through resident bills so they can see usage," she said. "We also communicate building features through resident portals, various ActiveBuilding communications and emails from the leasing office."
Data: the underpinning of apartment sustainability
With apartment sustainability initiatives come the need for adequate telecommunications bandwidth. Smart home technologies like Nest thermostats and other remote access features depend on data and a solid WIFI connection, and communities should plan ahead for technology's future.
In a session on telecom, one panelist encouraged developers to better communicate with construction teams so enough space is allocated for current and future equipment. Builders tend to stash technology in small spaces, which often isn't enough.
"No one thought about data and outlet ports," said Gail Corder, Ancillary Services Manager at Fairfield Residential. "Make sure you communicate with design and construction teams so that your needs for data are considered."
Data is the underpinning of sustainability and utility savings, said TGM Chief Operating Officer Zach Goldman.
Without clear energy consumption data, and the ability to benchmark this data and bill accurately from it, sustainability cannot succeed.
When residents and operators understand the numbers, it's then that multifamily gets its legs, adapts and embraces change.
"We are all focused on data," Goldman said. "We try to ensure our data is robust, and more importantly, accurate. Technology is certainly helpful. You need people who will take that data and move forward with action, drive bottom line NOI savings."
Greystar has become a proven leader in sustainability and innovation. A big reason, says Director of Global Sustainability Lexie Goldberg, is providing the proper degree of transparency and visibility to residents, owners, operators and investors.
Investing in sustainability has evolved over the past five years, with sustainability initiatives now commonly discussed with investors. The company showcases how it reaches goals, and frequently pauses for reassessment to ensure the efforts are on course.
Gathering data helps Greystar identify opportunities that are shared with those who can ultimately control utility usage at properties.
For its efforts, Greystar earned RealPage's first Utility Management Advisory 2020 Sustainability and Innovation Award.
The hunt is on for efficiencies, education
It's not all about the numbers. Even physically walking a property with a focus on identifying savings opportunities can be useful. Pete Zadoretzky, who serves as Vice President of Sustainability at Bozzuto Management, has turned ENERGY STAR's® Treasure Hunt into a fact-finding mission at the company's 260+ communities.
Site teams that participate can follow a "treasure map" to inspect common area lighting, building envelopes, vacant units, models, rooftops and garages, looking for inefficiencies. The very first High-Performance Inspection, as dubbed by Bozzuto, paid off immediately.
"One thing we found is that we had a lighting retrofit throughout the common areas that was assumed to be comprehensive," Zadoretzky explained. "Going through this process we found that wasn't exactly the case."
Thus, the property wasn't saving as much as it could have been.
The hunt will continue well into multifamily's future, said Dan Winters, who heads GRESB in the Americas. "The metrics are there, and it's just a matter of knitting together the data ecosystem to identify and address the gaps."
But it takes village.
"You look both at an asset as a whole and what can be done within the residences," says Chris Acker, VP of Community Technology Services for Lennar Multifamily Communities. "I think the IoT is the initial attraction, but the reality is that you start looking at how to maintain the asset and work with residents to understand how they can be sustainable within the unit. Sustainability is more effectively approached as a whole rather than in bits and pieces."
The RealPage Sustainability Suite provides the most comprehensive set of integrated energy management and conservation-related tools and professional services on the market. Learn more about solutions that have helped uncover millions of dollars in savings while increasing portfolio values and reducing risk exposure.
Have a question about our products or services?
Learn how RealPage can provide innovative solutions and services for your rental real estate business.
RealPage Analytics Blog
RealPage News
LeaseLabs Marketing Blog
Propertyware Single-Family Blog
Resident Engagement Blog


text:
'use strict';

//Xtasklists service used to communicate Xtasklists REST endpoints
angular.module('xtasklists').factory('Xtasklists', ['$resource',
	function($resource) {
		return $resource('xtasklists/:xtasklistId', { xtasklistId: '@_id'
		}, {
			update: {
				method: 'PUT'
			}
		});
	}
]);

redpajama_set_name: RedPajamaGithub


I am a Windows user. I am a .net developer also. A beautiful day, I try to make a small project with Memcache. But the Memcache system need to be deploy on Linux OS. I can using virtual machine to deploy it with a lot of to do item, so boring. I feel lazy.
I just think about Docker – containerization platform. Awesome, I install Docker, pull the Memcache image from Docker Store, run it as container and … every thing is ready for me. I try with many other image and there are work properly. I have a new way when work cross OS.
I recommend you should try to use Docker (or Kubernetes) and feeling happy with it.
Next Post [Stackoverflow] What is the difference between String and string in C#?