About Nestlé Nutrition InstituteFeatures To Enhance Beauty.
The Nestlé Nutrition Institute (NNI) is a non for profit association based in Vevey, Switzerland, that shares leading science-based information and education with healthcare professionals, scientists and nutrition communities and stakeholders in all parts of the world through various forms of interaction. The Institute was established in 1981 with the goal to foster "Science for Better Nutrition" to help enhance the quality of people's lives all over the world.
We are one of the world's largest private publisher of nutritional information including the PubMed-listed Workshop Book series and Annales Nestlé. As well as acquiring information on a broad range of nutrition related topics, we also share information through the acclaimed scientific workshops and symposia we organize and host internationally. The aim of these events is to make its findings available worldwide and so presentations are also made available through online videos and publications. Access to this information is free on our website www.nestlenutrition-institute.org , and over 300,000 people have chosen to sign up as NNI members.
Education is an important part of our strategy to share knowledge and further global understanding of the impact of nutrition at every stage of life. Since 1981, we provide fellowships grants* to young clinicians and scientists across the globe, with a particular focus on professionals working in the area of maternal and child health and nutrition. We also provide accredited continuing education programs for a wide range of healthcare professionals including doctors, nurses and dieticians.
Sign up to enjoy free access the latest research in nutrition science. Enjoy our extensive publications library, easy to read infographics, exclusive videos from leading experts in the field and benefit from accredited continuing education programs.


Eleanor McEvoy was on the radio a couple of nights ago while I was in the bath. I had to stop playing with my rubber duck, such was the beauty of her songs. She was chatting away and playing in front of a live audience in Dublin - I was hooked. Ducks and hooks in the same paragraph, blimey. She's on tour in the UK in April. I really hope you can get along. I know I'll be at one of her gigs, to be sure.
Here's one of Eleanor's beguiling songs that says in three minutes what most of us (songwriters and non-songwriters alike) struggle to say in a lifetime.
She's new to me CC. I'm really excited about discovering more about her, and seeing her later in the Spring.
Love her accent - I imagine you warm to the Irish brogue. She reminds me a bit of Charlene Spiteri but maybe that's not what you want to hear. Not long until Spring now.
I do! I love it. No mid-Atlantic nonsense from her.


text:
\section{Introduction}
\input{sec-introduction}

\section{Related Work}\label{sec:relatedwork}
\input{sec-related-work}

\section{Learning object motion from rotation-compensated flow}\label{sec:ICCV19}
\input{sec-learning-object-motion-flow}

\section{Experiments}\label{sec:experiments}
\input{sec-experiments}

\section*{Declarations}

\subsection*{Funding}
This work was supported in part by the ANR grant AVENUE (ANR-18-CE23-0011) and Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy – EXC 2002/1 ``Science of Intelligence" – project number 390523135.

\subsection*{Conflicts of interest/competing interests}
The authors have no conflicts of interest to declare that are relevant to the content of this article.

\section{Learning object motion from synthetic training data}\label{sec:ECCV18}

\subsection{Results}\label{sec:exp-estimFlow}

Our main framework consists of two steps (1) compensating the observed optical flow for camera rotation, and (2) segmenting the resulting optical flow in to static background and independently moving objects.
Experiments presented here are based on the DAVIS~\citep{perazzi2016benchmark} data set and the MoCA~\citep{lamdouar2020betrayed} data set, that each raise a slightly different aspect onto the motion segmentation problem. Details are described in the following.

\begin{table}
\small
\centering
\begin{tabular}{l | c | cccccccccc}
\hline
\multicolumn{1}{c|}{} & & LMP & TMM & Ours & Ours* \\
\hline
& Supervised & \ding{51} & \ding{55} & \ding{51} & \ding{51} \\
& RGB & \ding{55} & \ding{55} & \ding{55} & \ding{55} \\
& Flow & \ding{51} & \ding{51} & \ding{51} & \ding{51} \\
& Multi frame & \ding{55} & \ding{55} & \ding{55} & \ding{55} \\
\hline
\multirow{3}{*}{$\mathcal{J}$} & Mean $\uparrow$ & 58.4 &  40.1 & \bf\color{nonerikblue}59.7 & \cellcolor{yellow!70}\bf{62.5} \\
& Recall $\uparrow$& 67.3 &  34.3 & \bf\color{nonerikblue}{69.6} & \cellcolor{yellow!70}\bf{73.8} \\
& Decay $\downarrow$ & ~5.6 & 15.2 & \bf\color{nonerikblue}{~4.3} & \cellcolor{yellow!70}\bf{~3.8} \\
\hline
\multirow{3}{*}{$\mathcal{F}$} & Mean $\uparrow$ & 58.4 & 39.6 & \bf\color{nonerikblue}{59.5} & \cellcolor{yellow!70}\bf{61.1} \\
& Recall $\uparrow$ & 66.0 & 15.4 & \bf\color{nonerikblue}{66.4} & \cellcolor{yellow!70}\bf{69.9} \\
& Decay $\downarrow$& ~7.9 & 12.7 & \cellcolor{yellow!70}\bf{~5.4} & \bf\color{nonerikblue}{~5.6} \\
\hline
$\mathcal{T}$ & Mean $\downarrow$ & 87.8 & \cellcolor{yellow!70}\bf 51.3 & \bf\color{nonerikblue}74.5 & 83.4\\
\hline
\end{tabular}
\caption[Motion segmentation: Comparison to other approaches using only motion cues on DAVIS.]{Motion segmentation: Comparison to other approaches using only motion cues on DAVIS {\it(train-val)}, i.e., without any appearance. Ours refers to the variant of our model using only motion cues and no appearance terms and Ours* denotes a motion-only upper bound, which uses ground truth segmentation for camera motion estimation. Best viewed in color (\colorbox{yellow!70}{\bf1st-best}, {\bf\color{nonerikblue} 2nd-best}).}
\label{tbl:davis_ablation}
\end{table}

\paragraph{DAVIS: Optical flow only} We compare our motion segmentation network with other methods that use optical flow as the only cue for segmentation. Table~\ref{tbl:davis_ablation} shows these results on DAVIS.
LMP~\citep{Tokmakov17} is a learning based approach trained on ground truth optical flow of FlyingThings3D. This approach relies on a simliar network architecture, but does not incorporate an explicit model for modeling geometrical concepts, e.g. the scene geometry and camera motion. TMM~\citep{bideau2016s}, on the contrary, compensates flow for camera rotation and attempts to segment a video by assigning translational motion models to different image regions in a probabilistic fashion. The exclusive usage of translational motion models however quickly leads to oversegmentations and fails to capture more complex motion patterns. While combining geometrical concepts such as perspective projection together with learned motion patterns, our approach improves over both these motion segmentation methods. The segmentation performance is measured using the $\mathcal{J}$-Mean score. We achieve an $\mathcal{J}$-Mean score of 59.7. The next best performing method is LMP resulting in an $\mathcal{J}$-Mean score of 58.4.  We compute an upper bound for our method (Ours* in Table~\ref{tbl:davis_ablation}) by masking out independently moving objects, with ground truth segments, for our camera motion estimation procedure. This masking procedure eliminates errors of our camera motion estimation due to `outliers' in optical flow, such as moving objects.

\begin{table*}
\centering
\begin{tabular}{l | c | ccc|ccccccccc}
\hline
\multicolumn{1}{c|}{} & & SegI & MG & CIS & COD & COSNet & MATNet & Ours & Ours+Temp\\
\hline
\multirow{3}{*}{} & Supervised &  \ding{55} & \ding{55} &\ding{55} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51}\\
& RGB & \ding{55} & \ding{55} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{55} & \ding{55}\\
& Flow &  \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{55} & \ding{51} & \ding{51} & \ding{51}\\
& Multi-frame &  \ding{51} & \ding{51} & \ding{51} & \ding{55} & \ding{55} & \ding{51} & \ding{55} & \ding{51}\\
\hline
$\mathcal{J}$ & Mean $\downarrow$ & {\bf68.6} & 63.4 & 49.4 & 44.9 & 50.7 & \bf\color{nonerikblue}64.2 & 58.3 & \colorbox{yellow!70}{\bf65.8}\\
\hline
\multirow{6}{*}{\rotatebox[origin=c]{90}{Success Rate}} & $\tau = 0.5$ &  {\bf 77.2} & 74.2 & 55.6 & 41.4 & 58.8 & \bf\color{nonerikblue}71.2 & 64.5 &  \colorbox{yellow!70}{\bf72.7}\\
& $\tau = 0.6$ &  {\bf71.7} & 65.4 & 33.0 & 53.4 & \colorbox{yellow!70}{\bf67.0} & 46.3 & 58.0 & \bf\color{nonerikblue}65.2\\
& $\tau = 0.7$ & {\bf 62.3} & 52.4 & 32.9 & 23.5 & 45.7 & \colorbox{yellow!70}{\bf59.9} & 49.8 & \bf\color{nonerikblue}53.8\\
& $\tau = 0.8$ & {\bf 46.4} & 35.1 & 17.6 & 14.0 & 33.7 & \colorbox{yellow!70}{\bf49.2} & 36.2 & \bf\color{nonerikblue}38.9\\
& $\tau = 0.9$ & {\bf 25.5} & 14.7 & 3.0 & 5.9 & 16.7 & \colorbox{yellow!70}{\bf24.6} & 16.6 & \bf\color{nonerikblue}17.3\\
& $SR_{mean}$ & {\bf 56.6} & 48.4 & 31.1 & 23.6 & 41.7 & \colorbox{yellow!70}{\bf54.4} & 45.0 & \bf\color{nonerikblue}49.6\\
\hline
\end{tabular}
\caption[Motion segmentation: Comparison to state-of-the-art motion segmentation methods on MoCA.]{Motion segmentation: Comparison to state-of-the-art motion segmentation methods on MoCA. Methods we compare against from left to right:~\citep{Lamdouar21,yang2021self,yang2019unsupervised,lamdouar2020betrayed,Lu_2019_CVPR,zhou2020matnet}. {\bf Bold} indicates best among all methods, while \colorbox{yellow!70}{\bf1st-best} and {\bf\color{nonerikblue} 2nd-best} represent the best and second best within the supervised methods. Best viewed in color.}
\label{tbl:MoCA}
\end{table*}

\paragraph{MoCA: Optical flow only} Data sets like MoCA focus in particular on the segmentation of objects that can only be robustly recognized based on their unique motion. Where as most data sets for moving object segmentation combine several cues (motion and appearance) that are helpful for recognizing moving objects, this data set highlights the relevance of motion. Thus MoCA allows to evaluate the strengths of motion models in isolation.
It is not surprising that appearance cues are rather weak in cases of camouflage, therefore methods based on RGB frames only (e.g. COSNet~\citep{Lu_2019_CVPR}) show a weak performance in these settings (see Table~\ref{tbl:MoCA}). On more general data set like the DAVIS, those methods show a superior performance among all other methods and achieve a similar segmentation quality as MATNet~\citep{zhou2020matnet} (Table~\ref{tbl:davis_numerical_results}).

Our approach taking a single optical flow frame (compensated for camera rotation) as input, performs comparable to other supervised approaches. A simple post-processing step - convolution with a 3D Gaussian filter and frame-wise application of a dense CRF, eliminates temporal instabilities (Ours+Temp in Table~\ref{tbl:MoCA}). Among all methods SegI~\citep{Lamdouar21} shows best results on MoCA, on DAVIS their performance falls rather short due to their lack of a strong appearance model. SegI combines multiple ConvNets where each of them encode a flow frame together with a transformer network without taking RGB frames into consideration. The model is trained on synthetically generated data, thus can be considered as unsupervised. In contrast, our approach was trained using rotation compensated flow frames estimated from the synthetic dataset {\it FlyingThings3D}.

\begin{table*}
\small
\centering
\begin{tabular}{l | c | ccc|cccc|cccc}
\hline
\multicolumn{1}{c|}{} & & SFL & COSNet & MATNet & LMP+App & FSEG & LVO & Ours+App & CIS & ARP & SegI \\
\hline
& Supervised & \ding{68} & \ding{68} & \ding{68} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{55} & \ding{55} & \ding{55} \\
& RGB & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{55} \\
& Flow & \ding{51} & \ding{55} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} \\
& Multi-frame & \ding{55} & \ding{55} & \ding{51} & \ding{55} & \ding{55} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{51} \\
\hline
\multirow{3}{*}{$\mathcal{J}$} & Mean $\uparrow$ & 67.4 & 80.5 & \bf82.4 & 70.0 & 70.7 & \bf\color{nonerikblue}72.2 & \colorbox{yellow!70}{\bf73.5} & 71.5 & 76.2 & 67.8 \\
& Recall $\uparrow$ & 81.4 & 94.0 & \bf94.5 & \bf\color{nonerikblue}85.0 & 83.5 & 82.4 & \colorbox{yellow!70}{\bf85.5} & 86.5 & 91.1 & - \\
& Decay $\downarrow$ & 6.2 & \bf0.0 & 5.5 & 1.3 & 1.5 & \colorbox{yellow!70}{\bf0.1} & \bf\color{nonerikblue}1.2 & 9.5 & 7.0 & - \\
\hline
\multirow{3}{*}{$\mathcal{F}$} & Mean $\uparrow$ & 66.7 & 79.4 & \bf80.7 & 65.9 & 65.3 & \bf\color{nonerikblue}67.5 & \colorbox{yellow!70}{\bf68.9} & 70.5 & 70.6 & - \\
& Recall $\uparrow$ & 77.1 & \bf90.4 & 90.2 & \bf\color{nonerikblue}79.2 & 73.8 & 75.4 & \colorbox{yellow!70}{\bf79.6} & 83.5 & 83.5 & - \\
& Decay $\downarrow$ & 5.1 & \bf0.0 & 4.5 & 2.5 & \bf\color{nonerikblue}1.8 & 2.7 & \colorbox{yellow!70}{\bf1.4} & 7.0 & 7.9 & - \\
\hline
\end{tabular}
\caption[Motion segmentation: Comparison to state-of-the-art motion segmentation methods on DAVIS.]{Motion segmentation: Comparison to state-of-the-art motion segmentation methods on DAVIS2016. We group approaches according their training strategy: supervised and trained on the DAVIS training split (\ding{68}), supervised and trained on other segmentation data sets (\ding{51}) and unsupervised methods (\ding{55}). Methods we compare against from left to right:~\citep{Cheng_ICCV_2017,Lu_2019_CVPR,zhou2020matnet,Tokmakov17,fusionseg,Tokmakov17ICCV,yang2019unsupervised,koh2017primary,Lamdouar21,yang2021self}. {\bf Bold} indicates best among all methods, while \colorbox{yellow!70}{\bf1st-best} and {\bf\color{nonerikblue} 2nd-best} represent the best and second best within the supervised methods. Best viewed in color.}
\label{tbl:davis_numerical_results}
\end{table*}

\paragraph{DAVIS: Optical flow + Appearance} Where our main contribution lays in a novel approach to learn to segment moving object based on optical flow only, we incorporate here appearance information similar to LVO~\citep{Tokmakov17} and compare to segmentation approaches that consider both - appearance as well as motion information (Table~\ref{tbl:davis_numerical_results}).
Within the group of supervised approaches our approaches shows best performance in terms of mean/recall $\mathcal{J}$ and $\mathcal{F}$. Where as ours and LVO integrate appearance cues in a similar manner, these approaches differ in the way how object motion cues are learned. LVO learns object motion patterns directly from optical flow, where as we first disentangle camera rotation and translation before segmenting independently objects. Ablation studies analyze the usefulness of this disentanglement in further detail. Within the unsupervised approaches ARP~\citep{koh2017primary}, which is a non learning based approach, reaches highest performance. Due to multiple iterations over the entire video this approach is computationally expensive as mentioned in~\citep{yang2021self}.
Among all methods MATNet reaches highest accuracy in terms of mean $\mathcal{J}$ and $\mathcal{F}$. One reason might lay in their training strategy, which makes use of the DAVIS training set (indicated with \ding{68}).

A qualitative comparison with the best performing methods is shown in Figure~\ref{fig:results-comparison}. Our results based on optical flow only and based on optical flow in combination with appearance are shown in the last two rows of this figure. These two rows in particular highlight the complementarity of motion and appearance cues. We miss the hiker's foot when relying on motion alone (Ours), since it is not moving. However, while integrating motion with appearance, we segment the entire object accurately.
ARP, the strongest method among unsupervised approaches, relies on segmenting the primary object(s) in a video and and comes with a noticable bias towards the object's appearance. In many cases such a strong appearance model is advantageous, however can lead to erroneous segmentations in other cases. For example, it only segments a part of the car (Figure~\ref{fig:results-comparison}: 2nd column from the right), which moves from the darker (shadow) area to the brighter (sunny) region.), as it matches the primary object in appearance. Our method that extracts geometrical information from optical flow and integrates learned objectness cues is capable of overcoming these types of failure cases.

\subsection{Ablation study}
\label{sec:netvariants}

\paragraph{Network variants} We trained four variants of our motion segmentation network, with: (1) ground truth optical flow, (2) the ground truth flow after removing ground truth camera rotation, i.e., with rotation compensated-flow fields, (3) estimated optical flow field using PWC-Net~\citep{sun2018pwc}, and (4) estimated ground truth flow compensated with ground truth camera rotation, i.e., estimated rotation compensated-flow field.
Table~\ref{table:different-trainingData} shows the analysis with these four variants. Training and testing with ground truth optical flow (original: gt flow or compensated: gt t-flow) is significantly better than using estimated optical flow. Segmentation accuracy is about 20\% higher on the FT3D test set for ground truth, compared to estimated optical flow. Training with rotation-compensated optical flow consistently leads to improved quality of the final segmentation, e.g., 90.68\% vs. 93.23\%, which supports the idea behind our method. Learning can be significantly simplified, if we are able to efficiently incorporate knowledge about physical concepts into the process of moving object segmentation.
A direct comparison in terms of segmentation quality between using the original optical flow as input instead of the rotation-compensated optical flow is shown in Figure~\ref{fig:example-parkour}.

\begin{table}[b]
\begin{center}
\small
 \centering
\begin{tabular}
{llc}
\toprule
trained with... & tested with... & {angle+magnitude} \\
\midrule
gt flow & gt flow &  90.68 \\ \addlinespace[0.02cm]
gt t-flow & gt t-flow &  ~\bf93.23 \\ \addlinespace[0.02cm]
PWC flow & PWC flow &  77.18 \\ \addlinespace[0.02cm]
PWC t-flow & PWC t-flow & \bf78.69  \\ \addlinespace[0.02cm]
\bottomrule
\end{tabular}
\end{center}
\caption[Ablation study: Network variants.]{{Ablation study: Network variants.} We trained four networks using flow angle and magnitude with: the provided ground truth optical flow of FT3D~\citep{mayer2016large} (gt flow), ground truth optical flow after subtracting ground truth camera rotation (gt t-flow), estimated optical flow using~\citep{sun2018pwc} (PWC flow), and estimated optical flow after subtracting ground truth camera rotation (PWC t-flow). Segmentation accuracy is measured on the FT3D test set with intersection over union (IoU) scores.}
\label{table:different-trainingData}
\end{table}

\begin{figure}
\begin{center}
\includegraphics[width=0.95\columnwidth]{parkour.pdf}
   \caption[Comparison of motion segmentation results based on the original and the rotation-compensated flow field.]{{\bf Ablation study: Comparison of motion segmentation results based on the original and the rotation-compensated flow field.} Top row: motion segmentation with the original flow field that includes camera rotation, translation and object motion. Bottom row: motion segmentation based on \textit{rotation-compensated flow field}. Note that the angle field (middle) of the rotation-compensated flow is entirely depth independent. The angle field is fully determined by the translational camera motion and object motion. In this example one can observe a clear z-motion of the camera, which is shown by the rainbow pattern. The angle field of the original flow containing both camera rotation and translation is depth dependent (top row, middle image). This angle field clearly shows discontinuities in angle at the wall, which is due to significant changes in depth and {not} because of independent object motion.}
   \label{fig:example-parkour}
\end{center}
\end{figure}

\paragraph{Training on flow angle only versus angle+magnitude} As discussed in Section~\ref{sec:motionfield}, rotation-compensated flow comprises all the information about independent object motion and the scene structure (depth). In this context, two interesting questions to tackle are: \textit{how well can one extract information about independent object motion from the angle alone}, and \textit{does including the flow magnitude (training the network on the full optical flow) improve motion segmentation?}. We show this analysis in Table~\ref{table:angleVangle+magnitude}, with further variants of our network. Using angle and magnitude together (angle+magn in the table) leads to the best performance. However, note that we achieve reasonable segmentation quality even when using the flow angle alone. The network trained on ground truth optical flow adapts very poorly to estimated optical flow, with the segmentation accuracy dropping from 93.23\% to 24.44\% for the angle+magn variant.

\begin{table}
\begin{center}
\small
 \centering
\begin{tabular}
{ll*{2}{c}}
\toprule
trained with... & tested with... & {angle} & {angle+magn} \\
\midrule
gt t-flow & gt t-flow  & 77.47 &  93.23 \\ \addlinespace[0.02cm]
gt t-flow & PWC t-flow  & 24.06 & 24.44 \\ \addlinespace[0.02cm]
PWC t-flow & PWC t-flow  & 77.79 & \bf 78.69  \\ \addlinespace[0.02cm]
\bottomrule
\end{tabular}
\end{center}
\caption[Ablation study: Training with angle vs angle and magnitude.]{{Ablation study: Training with angle vs angle and magnitude.} We trained four variants of our segmentation network with: (1)~angle of the rotation-compensated flow of FT3D, (2)~angle and magnitude of the rotation-compensated flow of FT3D (angle+magn), (3)~angle of the estimated rotation-compensated flow, and (4)~angle and magnitude of the estimated rotation-compensated flow. We show consistently better performance by including magnitude. The performance is the worst when the network is trained on the angle of the rotation-compensated ground truth flow. Here, the noise in angle leads to a very significant drop on estimated optical flow data. Segmentation accuracy is measured on the FT3D test set with intersection over union (IoU).}
\label{table:angleVangle+magnitude}
\end{table}

\paragraph{Rotation estimation via likelihood maximization}

We show results on the Sintel data set (Table~\ref{tbl:sintel_ablation}), and compare our new likelihood optimization procedure with~\citep{bideau2016s}. The ground truth optical flow and focal length is provided, so an accurate estimate of the camera's rotation is possible.
Our camera rotation estimation based on maximizing the flow likelihood shows consistently better results on the Sintel data set. More importantly, the performance gap gets significant when using estimated flow as input for camera motion estimation. Since our proposed optimization approach incorporates an explicit noise model, it is significantly more robust to noisy flow data.

\begin{table}
\small
\centering
\begin{tabular}{l  c  c}
\toprule
 & TMM & Ours \\
 \midrule
gt-flow & 0.08 / 0.22 / 0.02 & 0.03 / 0.06 / 0.01 \\
PWC-flow & 0.13 / 0.34 / 0.04 & 0.05 / 0.11 / 0.03 \\
\bottomrule
\end{tabular}
\caption[Ablation study: Camera rotation estimation.]{{Ablation study: Camera rotation estimation.} Avg.\ yaw/pitch/roll error in degrees between two consecutive frames.
\textit{gt-flow, PWC-flow:} To evaluate rotation estimation we used ground-truth segmentation masks to weight the optim.\ loss. Thus, errors in the segmentation procedure are not propagated throughout the video.
}
\label{tbl:sintel_ablation}
\end{table}

\begin{figure*}
\begin{center}
\includegraphics[width=1.0\textwidth]{results.pdf}
   \caption[Qualitative segmentation results.]{{\bf Qualitative segmentation results.} Qualitative segmentation results on the DAVIS data set, showing a comparison with three other best performing methods. Ours-final denotes our complete method and Ours the variant based on motion cues alone.}
   \label{fig:results-comparison}
\end{center}
\end{figure*}

\subsection{The Motion Field: A Geometrical Analysis}\label{sec:motionfield}
The motion field captures pixel displacements between two consecutive frames. Displacements arise typically due to one of the following factors: (1)~a moving camera, (2)~one or more objects moving in the 3D world. These pixel displacements depend not only on the speed of objects or the camera, but also the scene geometry.

As an example to illustrate the different factors that influence the formation of the motion field, let's consider the ``goat'' sequence from the DAVIS data set (Figure~\ref{fig:overview}). Based on the original flow field it is hard to estimate which pixels belong to the moving object and which belong to static background. The direction of the flow in the background region differs significantly from the flow describing the motion of the rocks in the foreground region (motion direction is color encoded). However, neither the background nor the rocks are moving differently in the 3D world. To detect objects that are actually moving independently in 3D it is necessary to decompose the observed motion field.
We formalize these observations and review the geometrical construction of the motion field.

\subsubsection{Motion field}
Let $[U,V,W]$ be the parameters describing the camera translation and $[A,B,C]$ the parameters describing camera rotation\footnote{The rotation parameters are often referred to as pitch, yaw and roll.} along the x, y and z axes respectively. Let $f$ be the camera's focal length and $Z$ the relative scene depth at a pixel location $(x,y)$. In this setting, the motion vector $\vec{v}$ due to camera motion is given by:
\begin{align}
\vec{v} &= \vec{v}_r + \vec{v}_t =
\begin{pmatrix}
  u_r  \\
  v_r
 \end{pmatrix} +
 \begin{pmatrix}
  u_t  \\
  v_t
 \end{pmatrix},\\
 &=
 \begin{pmatrix}
  \frac{A}{f}xy  - Bf - \frac{B}{f}x^2 + Cy \\
  Af + \frac{A}{f}y^2 - \frac{B}{f}xy - Cx
 \end{pmatrix}
 +
 \begin{pmatrix}
  \frac{-fU+xW}{Z}\\
 \frac{-fV+yW}{Z}
 \end{pmatrix},
 \label{eqn:flowdecomp}
\end{align}
where $\vec{v}_r$ and $\vec{v}_t$ represent motion field vectors corresponding to camera rotation and translation respectively. Equation~(\ref{eqn:flowdecomp})\footnote{In fact, this equation is an approximation, and only holds if the rotation angles are small~\citep{longuet1980interpretation}. To obtain the exact rotational flow field one has to transform the 2D image points to 3D using perspective projection equations, rotate the points according to the camera's rotation in 3D, backproject them onto the 2D image plane, and then measure the displacement.} highlights an important properly, namely that the flow due to camera rotation is only determined by the camera rotation parameters and the camera's focal length. {\it  The flow due to camera rotation is independent of the scene depth.}  One can subtract this rotational motion component at each pixel to obtain a rotation-compensated flow field.

\subsubsection{Rotation-compensated motion field}\label{sec:motionfield-anglefield}
As shown in the flow equation (\ref{eqn:flowdecomp}), the rotation-compensated flow field $\vec{v}_t$ is determined by the translational camera motion $[U,V,W]$, and the scene depth $Z$. It comprises all the relevant information about the scene geometry, unlike the rotational component $\vec{v}_r$, which is independent of the scene geometry. The magnitude of the rotation-compensated flow is inversely related to scene depth, i.e., regions further away from the camera have small translational flow magnitude, and those closer to the camera have larger magnitudes. {\it The direction of $\vec{v_t}$ (flow angle) however does not depend upon the scene depth}:
\begin{align}
\theta=
\begin{cases}
  \arccos{(xW - fU)}, & \text{if}\ (yW - fV)>0, \\
  2\pi-\arccos{(xW - fU)}, & \text{otherwise.}
\end{cases}
\label{eq:anglefield}
\end{align}

Figure \ref{fig:motion-vec-def} pictures the computation of the flow angle $\theta$ at pixel locations $(x,y)$, leading to an angle field as shown on the right. Where as Figure~\ref{fig:motion-vec-def} pictures the angle field of pure camera translation, Figure~\ref{fig:overview} shows an angle field of a scene with camera translation and object motion. Here, independently moving objects, can be observed as discontinuities in angle. The angle of the rotation-compensated flow alone is independent of the scene geometry, thus independently moving objects stand out due to their different direction.

\begin{figure}[h]
\begin{center}
   \includegraphics[width=0.99\columnwidth]{flowVec-def.pdf}
   \caption{{\bf Translational motion field vector.} Left: motion field vector $\vec{v}_t$ at pixel position $(x,y)$. Right: color coding of the angle field $\theta(x,y)$ at each pixel location for the case of camera translation along the optical axis, i.e. $[U,V,W]=[0,0,1]$.}
   \label{fig:motion-vec-def}
\end{center}
\end{figure}

\begin{figure*}
\begin{center}
    \subfigure[Translational (observed) flow vector $\vec{o}_t$ at pixel location $(x,y)$.]
    {
        \includegraphics[width=0.2\textwidth]{likelihood_a.pdf}
    }\hspace{2em}
    \subfigure[Observed optical flow vector $\vec{o}_t$, is a noisy observation of the motion field vector $\vec{v}_t$: $\vec{o_t} = \vec{v}_t + \vec{n}$.]
    {
        \includegraphics[width=0.2\textwidth]{likelihood_b.pdf}
    }\hspace{2em}
    \subfigure[To compute the flow likelihood, we integrate over the unknown motion magnitude of the motion field vector $\vec{v}_t$.]
    {
        \includegraphics[width=0.2\textwidth]{likelihood_c.pdf}
    }\hspace{2em}
    \subfigure[Probability distribution over inverse depth.]
    {
        \includegraphics[width=0.2\textwidth]{invDepth.pdf}
        \label{fig:invDepth}
    }
   \caption[The flow likelihood]{{\bf Flow likelihood.} (a)-(c): computation of the probability $p(\vec{n}$) at pixel location $(x,y)$. (d): probability distribution over inverse depth. The flow likelihood is maximal, when the observed flow vector $\vec{o}_t$ and the motion field vector $\vec{v}_t$ point into the same direction with similar magnitude.}
   \label{fig:likelihood-noise}
\end{center}
\end{figure*}

\subsection{The Right Spin: Camera Motion Estimation}\label{sec:cam-estimation}
To rectify the observed optical flow field for camera rotation, we require an accurate estimate for rotation. How can we obtain a good estimate of the camera rotation and the translational motion direction that together best explain the observed motion field? Towards finding an answer to this question, we derive a novel maximum likelihood approach that aims at finding the rotation $[A,B,C]$ such that the likelihood of the resulting translational flow field is maximized.
To this end, we derive a new flow likelihood function incorporating a model for the optical flow's noise as well as a prior distribution over the {\em inverse scene depth}.

In the following, we first introduce the new flow likelihood (Section~\ref{sec:flow-likelihood}).
We then describe how camera motion parameters are estimated by maximizing this new likelihood function.

\subsubsection{Likelihood of the translational motion field}\label{sec:flow-likelihood}
 Let $\vec{o}_t$ be the observed translational flow vector, e.g., flow estimated with \citep{sun2018pwc}, at the pixel position $(x,y)$. Let the translational 3D motion direction of the camera $[U,V,W]$ be a unit vector. The three translational camera parameters $[U,V,W]$ and the pixel position $(x,y)$ define the direction of a motion field vector on the image plane .  As derived in~\citep{bideauCVPR2018}, the probability of observing  $\vec{o}_t$ at $(x,y)$ given a motion direction $[U,V,W]$ is given by:
\begin{align}
p(\vec{o}_t & \mid U, V, W, x, y)\nonumber\\
&=\int_0^{\infty} p(\vec{n})\;p_r(r \mid U, V, W, x, y)  \,dr, \label{eq:likelihood}
\end{align}
where $r$ denotes the magnitude of a motion field vector and $\vec{n}$ is the optical flow's noise.
This likelihood function depends on the distribution over the optical flow's noise $p(\vec{n})$ as well as the distribution over motion field magnitudes $p_r$. Figure~\ref{fig:likelihood-noise} pictures the computation of $p(\vec{n})$. Modeling the probability distribution over flow magnitudes is challenging, since those depend on the camera's translational motion direction $[U,V,W]$, the pixel location as well as the scene depth at that location. \citep{bideauCVPR2018} model $p_r$ by assuming that the motion field magnitude $r$ is independent of the flow direction $[U,V,W]$. However this often does not lead to accurate estimations, especially in the case of strong z-motion (forward motion). Here, motion field magnitudes close to the focus of expansion are near zero and the motion vectors farther away from the focus of expansion show larger magnitudes, thus the motion field magnitude is clearly dependent upon the camera's motion direction.

Next, we present a new way of modeling the distribution over motion field magnitudes $p_r$ that alleviates this problem.

\begin{comment}
\paragraph{From flow magnitudes to inverse depth.}
Given the motion field of a pure translational motion, from the perspective projection equations~\citep{bruss1983passive}, we can derive the motion field components $u_t$ and $v_t$ as:
\begin{align}
u_t&=\frac{-fU + xW}{Z}, & v_t&=\frac{-fV + yW}{Z}.
\end{align}
Then the motion field magnitude $r$ is:
\begin{align}
r&=\sqrt{u_t^2+v_t^2},\\
&=\frac{1}{Z} \cdot \sqrt{(-fU+xW)^2+(-fV+yW)^2},\\
&=\frac{1}{Z} \cdot g(f,x,y,U,V,W), \label{eqn:rgrelation}
\end{align}
where $g$ is a function that controls all aspects of the magnitude that are {\em not} related to depth.
\end{comment}

\paragraph{Distributions over flow magnitudes}
We express the motion field magnitudes in terms of inverse depth $\frac{1}Z{}$ and $g(\cdot)$, which is a function comprising all aspects of the magnitude that are {\em not} related to depth,
\begin{align}
r&=\sqrt{u_t^2+v_t^2},\nonumber\\
&=\frac{1}{Z} \cdot g(f,x,y,U,V,W). \label{eqn:rgrelation}
\end{align}
Given this reformulation of the magnitude $r$, we can determine the {\em induced distribution} over motion field magnitudes, given the distribution over inverse depths. We aim to compute $p_r(r\mid g(f,x,y,U,V,W))$ through
$p_{\frac{1}{Z}}(\frac{1}{z})$, which is the distribution over inverse depth.
Using the relation between $r$ and $g(\cdot)$ from Eq.~\ref{eqn:rgrelation}, we can rewrite $p_r(r\mid g(\cdot))$ as follows
\begin{equation}
\label{ref:change1}
p_r(r\mid g(\cdot)) = \frac{p_{\frac{1}{Z}}(\frac{r}{g(\cdot)})}{g(\cdot)}.
\end{equation}
This is effectively just a change of units.
Expressing the distribution over flow magnitudes in terms of the distribution over inverse depth however brings a significant advantage. This formulation effectively factors motion direction $(U,V,W)$, focal length $f$ and scene depth into the function $g(\cdot)$, and the distribution over depth can be modeled without relying on these dependencies that require making further approximations.

\paragraph{Flow likelihood}
Following prior derivations, the flow likelihood function (Eq. \ref{eq:likelihood}) can be expressed by the distribution over inverse depth, instead of flow magnitudes:
\begin{align}
p(\vec{o_t} & \mid U, V, W, x, y)\nonumber\\
&=\int_0^{\infty} p(\vec{n})\;p_r(r \mid g(\cdot))  \,dr \nonumber\\
&= \int_0^{\infty} p(\vec{n})\;\frac{p_{\frac{1}{Z}}\left(\frac{r}{g(\cdot)}\right)}{g(\cdot)}  \,dr. \label{eq:final-likelihood}
\end{align}
The key advantage of this is that while flow magnitudes are not independent of the motion direction, the inverse depths are, and thus the model is more realistic.

\subsubsection{Camera motion estimation via likelihood maximization}\label{sec:cam-est-trans}
In Section \ref{sec:flow-likelihood}, we have derived a new likelihood function of an observed optical flow vector $\vec{o}$.
Our goal is now to find a camera rotation $(A, B, C)$ and translational camera motion direction $(U,V,W)$, such that the flow likelihood is maximal or alternatively the negative log-likelihood is minimal.
Recall $\vec{o}_t$ is the observed translational flow vector after subtracting the flow $\vec{v}_r$ due to camera rotation:
\begin{align}
    \vec{o}_t = \vec{o} - \vec{v}_r(A,B,C).
\end{align}
Given the rotation compensated flow, we minimize the negative log-likelihood as follows:
\begin{align}
&A^*,B^*,C^*,U^*,V^*,W^*\nonumber\\
&= \operatorname*{arg\,min}_{A,B,C,U,V,W} \sum -\log(p(\vec{o_t}\mid U, V, W, x, y)).
\end{align}
Local minima are a concern when solving this optimization problem, especially in cases of noisy optical flow, inaccurate estimates of independently moving objects or complex scene geometry. To reduce this risk, we initialize the optimization using three different starting points: (1) camera rotation and translation estimate of the previous frame, (2) camera rotation estimate weighted by depth estimate of the previous frame and the translation estimate of the previous frame, and (3) camera rotation estimate weighted by depth estimate of the previous frame and the translation estimate of the previous frame in the opposite direction.
The first initialization is a good assumption if the camera motion is approximately constant.
Initialization (2) and (3) incorporate depth information. The apparent motion of areas far away is mainly influenced by the camera's rotation and not the camera's translation (see Figure~\ref{fig:overview-flowields}), thus knowing the depth helps to correctly disentangle flow due to camera rotation and flow due to translation.

During the optimization each pixel is weighted using learned, soft object motion masks of the previous frame, that evolve over time - thus the influence of moving objects is suppressed due to a low weight. The following Section describes how object motion masks are learned while pertaining important geometric information.

\begin{figure}
\begin{center}
\includegraphics[width=0.8\columnwidth]{CVPR19_likelihood_a-6.pdf}
   \caption[A 3D lookup table for the flow likelihhood.]{{\bf Lookup table picturing flow likelihood values.} Our new flow likelihood addresses the challenge of estimating the camera's motion in the presence of noisy optical flow. The color \textit{red} indicates high likelihood values, \textit{dark blue} indicates low likelihood values. The lower the angle difference $\Delta \theta$ between the vectors $\vec{o}_t$ and $\vec{v}_t$, the higher the likelihood. Note that for very small flow magnitudes $m$ the flow likelihood is almost the same regardless $\Delta \theta$. This is an important consequence of our model, indicating the unreliability of the flow direction in case of near zero magnitudes.}
   \label{fig:lookup}
\end{center}
\end{figure}

\begin{figure*}
\centering
\subfigure[video frame]
{
	\includegraphics[height=0.157\textwidth]{forest_00001.png}
	\includegraphics[height=0.157\textwidth]{traffic_00001.png}
	\includegraphics[height=0.157\textwidth]{parkour.jpg}
	\includegraphics[height=0.157\textwidth]{goat.jpg}
    \label{fig:overview-flowields-frame}
}\\
\subfigure[optical flow]
{
	\includegraphics[height=0.15\textwidth]{flow-orig.png}
	\includegraphics[height=0.15\textwidth]{traffic-flow-orig.png}
	\includegraphics[height=0.15\textwidth]{parkour-flow.png}
	\includegraphics[height=0.15\textwidth]{goat-flow.png}
    \label{fig:overview-flowields-flow}
}\\
\subfigure[rotation compensated optical flow]
{
    \includegraphics[height=0.15\textwidth]{flow-trans.png}
	\includegraphics[height=0.15\textwidth]{traffic-flow-trans.png}
	\includegraphics[height=0.15\textwidth]{parkour-flow-trans.png}
	\includegraphics[height=0.15\textwidth]{goat-flow-trans.png}
    \label{fig:overview-flowields-trans-flow}
}\\
\subfigure[depth estimate]
{
	\includegraphics[height=0.15\textwidth]{forest-depth.png}
	\includegraphics[height=0.15\textwidth]{traffic-depth.png}
	\includegraphics[height=0.15\textwidth]{parkour-depth.png}
	\includegraphics[height=0.15\textwidth]{goat-depth.png}
    \label{fig:overview-flowields-depth}
}
\caption[Flow, rotation compensated flow and the relative depth estimates.]{{\bf Flow, rotation compensated flow and the relative depth estimate.} We show sample videos from the data set Complex Background (video sequences: traffic, forest) as well as two sample videos from the Davis data set (video sequence: parkour, goat). A comparison of (b) and (d) shows how motion at distant is dominated by camera rotation. After subtracting of the camera's rotation the remaining flow magnitude in these areas is very small (light color). If the flow magnitude is small the motion direction is noisy. This can be seen in (e).}
\label{fig:overview-flowields}
\end{figure*}

\subsection{Object Motion Segmentation}\label{sec:learning-motion}
We build our segmentation framework on an effective model for motion segmentation, that learns object motion patterns from optical flow and segments a flow field into static background and moving objects~\citep{Tokmakov17}. Yet, this model does not incorporate any geometrical concepts.
As discussed earlier optical flow fields couple information about scene geometry as well as camera motion, making the judgment whether an object is moving challenging.
By introducing a simple pre-processing step we show, that the complexity of optical flow patterns is dramatically reduced.
Different from prior work, our network processes rotation compensated flow fields (angle + magnitude) to segment independently moving objects. Learning object motion based on pre-processed flow fields appears to be an easier task to learn.
While our network architecture is similar to~\citep{Tokmakov17}, we propose important modifications to the training procedure in the following.

\subsubsection{Incorporating geometric information into training} \label{sec:training}
The network follows the classical U-net architecture and is trained on estimated translational flow fields. During training, we first estimate optical flow using~\citep{sun2018pwc} on the FlyingThings3D data set~\citep{mayer2016large}. The ground truth camera rotation is provided and subtracted from the estimated flow to obtain a rotation-compensated flow field. This flow field is input to our network. The input has a size of $h \times w \times 3$. The third dimension denotes the flow expressed in terms of angle (represented as a unit vector) and magnitude.
A representation of the flow angle as unit vector instead of angles in degree avoids segmentation discontinuities at 0 degree (or $2\pi$ respectively). The normalized flow field and the flow's magnitude are concatenated and form the input to our network. An interesting question for training a network with rotation-compensated optical flow is, whether it is worthwhile to incorporate the magnitude into the training procedure. On the one hand
the flow magnitude can be a good indicator about the reliability of the flow angle~\citep{bideau2016s}, while on the other hand variation in larger magnitudes can be either due to variances in the scene depth or fast moving objects - thus including the magnitude might add rather misleading information. We take a closer look into this question as part of our ablation study in Section~\ref{sec:netvariants}.

\subsection{Implementation details}

To find the camera rotation and translational motion direction that best explains the observed optical flow field, we derived a new flow likelihood function (Section~\ref{sec:cam-estimation}). Details regarding parametrization are provided in the following.

The probability of the flow noise $p(\vec{n})$ is modeled as a multivariate normal distribution
$p(\vec{n}) \sim \mathcal{N}(\mu, \Sigma)$
and the inverse depth $p(\frac{1}{Z})$ as
an exponential distribution
$p(\frac{1}{Z}) \sim \textrm{Exp}(\lambda)$.
The noise covariance $\Sigma$ is assumed to be spherical and is measured using the ground truth flow of Sintel \citep{Butler:ECCV:2012} and the corresponding noisy estimate~\cite{sun2018pwc}.
We obtain $\Sigma=16.5\cdot10^{-5} I$, where $I$ is the identity matrix.
$\lambda$ is the rate parameter of the exponential distribution modeling the inverse depth, and is estimated using ground truth depths from Sintel.
We measured $\lambda=0.64$. The distribution over inverse depth can be seen in Figure~\ref{fig:invDepth}.

For computational efficiency the integral in Eq.~\ref{eq:final-likelihood} is approximated using a discrete sum over motion field magnitudes $r$.
Flow likelihood values are pre-computed and stored in a lookup table for efficiency (see Figure~\ref{fig:lookup}).
\subsection{Classical approaches}

\paragraph{Methods based on feature clustering}\label{ref:sec-literature-trajectory}
To capture motion information, typically point trajectories are either formed by tracked image features or dense optical flow.
Then trajectories sharing similar motion characteristics are grouped into coherent motion clusters describing the motion of a particular object~\citep{keuper2015motion,brox2010object,fragkiadaki2012video,ochs2011object,keuper2017,yan2006general,shen2018submodular,lezama2011track}.

These approaches vary in defining typical motion characteristics for clustering. Yan et. al \citep{yan2006general} propose to cluster trajectories based on geometric constraints (trajectories of the same motion lie in a manifold) and locality. In \citep{keuper2017,keuper2015motion} the segmentation problem is represented as a minimum cost multicut graph problem, where edge weights are computed from motion, position and color cues.

These trajectory based clustering approaches reach their limit if understanding of the scene structure is necessary to segment a moving object correctly. Trajectories perfectly represent long-term pixel displacements between a sequence of frames.
Pixel displacements however are a function of depth and motion. Thus trajectory based clustering methods often form clusters not only for independently moving objects, but also for objects at different depths. For instance if the camera is translating and rotating rocks close to the camera produce a very different flow pattern that the far away scene (see Figure \ref{fig:overview}), thus those two areas would form two separate clusters although neither the rock nor the far away scene is moving.

Methods based on occlusions~\citep{ogale2005motion,taylor2015causal} are subject to similar depth-related problems, since occlusions could be caused at depth boundaries as well as motion boundaries. A distinction is often not made.

\paragraph{Methods based on projective geometry} \label{ref:sec-literature-projective}
Projective geometry is an extension of the Euclidean and affine space and contains properties of perspective projection.
It is widely used as a mathematical formalism  to describe the geometry of cameras and its associated transformations \citep{torr1998geometric,zamalieva2014background,wang1994representing,ke2002robust,jin2008background,xiao2005motion,vidal2004unified,xu2018motion}.

Different from trajectory based clustering methods, motion segmentation approaches relying on projective geometry analyze the optical flow
between a pair of frames, grouping pixels into regions where flow is
consistent with motion models that are explainable by projective geometry \citep{torr1998geometric,zamalieva2014background,wang1994representing,ke2002robust,jin2008background,xiao2005motion,xu2018motion}.
Torr~\citep{torr1998geometric} develops a sophisticated probabilistic
model of optical flow, building a mixture model
that explains an arbitrary number of rigid components within the
scene. Interestingly, he assigns different types of motion models to
each object based on model fitting criteria.
Zamalieva et al.~\citep{zamalieva2014background} and Xun Xu et al.~\citep{xu2018motion} present a combination of methods that rely on both - projective geometry (homography estimation) and perspective projection (fundamental matrix estimation). The two methods have complimentary
strengths, and the authors attempt to select among the best
dynamically.

Methods relying on projective geometry perform well in cases of planar motion (motion obtained by a translating or rotating camera picturing a planar scene or a very distant scene, where effects of 3D parallax are negligible), however similarly to cluster based approaches these methods fall short in case of complex scene geometry.

Horn identified specific drawbacks of using projective geometry in such estimation problems and has argued that methods based directly on
perspective projection are less prone to overfitting in the presence of noise \citep{horn1999projective}.

\paragraph{Methods based on perspective projection} \label{ref:sec-literature-perspective}
Perspective geometry allows us to mathematically explain and model the process of how the three-dimensional world is projected on to a just two-dimensional image plane. Artists and scientists like Alberti, Brunelleschi, D\"urer and da Vinici studied effects of perspective projection about 500 years back in time~\citep{pirenne1952scientific}. These insights have made a significant contribution to current successes in computer vision.
One of the key aspects of perspective projection is the observation that two parallel lines (in the euclidean space) are transformed to two lines that intersect in the vanishing point at the horizon on the image plane.

It has been shown that motion segmentation approaches based on perspective projection \citep{irani1998unified,bideau2016s,bideauCVPR2018,narayana2013coherent,vidal2002segmentation,zhang2007moving, yang2021learning} are more accurate (in terms of model agreement to the physical world) than those based on projective geometry, since the latter omits certain constraints in modeling image transformations~\citep{horn1999projective,bideau2016s}. Having a model that is confirm with the physical world might be especially critical for tasks where interaction with the physical world is required in a second step such as in robotics or autonomous driving.

\subsection{Learning motion segmentation using convolutional neural networks}

\paragraph{Methods based on supervised learning}
Several approaches as~\citep{Tokmakov17,Tokmakov17ICCV,fusionseg,Cheng_ICCV_2017,dave2019towards,ranjan2019competitive,vertens2017smsnet, Mahadevan20BMVC, lamdouar2020betrayed, cheng2017segflow}
have explored the strength of deep neural networks to learn motion patterns of moving objects and to produce binary motion masks distinguishing whether a pixel belongs to a moving object or not. Most approaches propose a two-stream architecture~\citep{Tokmakov17ICCV,fusionseg,dave2019towards} to separately process motion and appearance.

\noindent Theses approaches learn motion patterns given the optical flow, the raw video frames or optical flow together video frames. Rather than following the true physics of image formation, convolutional neural networks are able to learn high level motion patterns of background motion and object motion.
This ability has the clear advantage of not being dependent upon technical camera parameters such as the focal length or image distortions due to various lens characteristics or constraints induced by technical parts of the camera (mechanical or electronic).

\paragraph{Methods based on self-supervised learning}
General concerns of deep-learning based approaches and in particular supervised approaches are overfitting to a particular type of object category that is likely to move~\citep{dave2019towards} and the lack of large amounts of training data.
To overcome the problem of limited training data, two straight forward approaches are either using synthetic training data~\citep{Tokmakov17,Tokmakov17ICCV} or relying on noisy estimates of the motion field~\citep{fusionseg} using other algorithms~\citep{sun2018pwc,ilg2017flownet,sun2010secrets}. However, both paths are still in need of large amounts of training data (although no additional manual annotations are required in these cases), this rises the need for self-supervised approaches~\citep{yang2021self, lu2019see, yang2019unsupervised, lai2020mast,gordon2019depth,Bideau_2018_ECCV_Workshops}.
Incorporating knowledge about the real world physics into the training procedure of a neural network is an alternative to various kinds of data augmentation approaches that is subject of current research~\citep{tung2019learning, yang2021learning}. Some of those ideas have been already successfully applied in context of self-supervised learning~\citep{gordon2019depth,Bideau_2018_ECCV_Workshops}.

Here, we propose a novel approach to the motion segmentation problem that specifically combines aspects of perspective projection and learns general object motion patterns.

redpajama_set_name: RedPajamaArXiv


text:
104 Tauri (m Tauri) é uma estrela na direção da Taurus. Possui uma ascensão reta de 05h 07m 26.68s e uma declinação de +18° 38′ 42.0″. Sua magnitude aparente é igual a 4.91. Considerando sua distância de 52 anos-luz em relação à Terra, sua magnitude absoluta é igual a 3.91. Pertence à classe espectral G4V.

Ver também
 Designação de Bayer
 Designação de Flamsteed
 Catálogo Henry Draper
 Catálogo Hipparcos

Estrelas
Objetos de Bayer
Constelação de Taurus

redpajama_set_name: RedPajamaWikipedia


Any Order that a court issues during the pendency of a Divorce or Paternity case, that is not a final Order of all issues, is a provisional Order. These can also be called "interlocutory", "preliminary", or "temporary" Orders.
A Provisional Order is meant to maintain the status quo as close to possible. Any time a divorce action begins, there are inherently a lot of life changes, especially if someone has moved out of the shared house, and there are children involved. This applies to Paternity cases as well, as oftentimes when the parties come to court is because their former romantic relationship has ended and they have to determine how to co-parent together.
In Divorce cases, the Dissolution Act specifically allows for such preliminary orders when a divorce case is filed. The court may issue an order for temporary spousal maintenance (formerly called alimony for spousal support), child custody and parenting time, and possession of property while the case is pending before the Court1. This is because cases, especially contested cases, take time to prepare and litigants often need at least some temporary resolution to matters in dispute immediately.
Because these issues may reasonably require multiple hearings to resolve, entering a provisional order in a paternity proceeding concerning parenting time and custody is quite appropriate while relevant issues are developed for resolution in a final hearing, if necessary, and a final order.
Provisional Orders, and requests for Preliminary Hearings, can be very useful in both divorce and paternity cases, where there is a lot of unknown, and the case could take several months to finally adjudicate, especially if a contested trial is needed. Provisional Orders can ensure that both parents have frequent and regular contact with the child during the case.
We hope that you have found this information to be helpful in understanding what provisional orders are, and how and when they are used in divorce and paternity cases. This is not intended to be legal advice. If you have questions or concerns about your specific case, CIYOU & DIXON, P.C. can help evaluate your specific case. This blog post was written by Attorney, Lori B. Schmeltzer.


(1) The University has established a number of subsidiary entities over which it exercises control. That control enables the University to govern the financial and operating policies of each entity for the benefit of the University in fulfilling its objects and conducting its functions. In the course of their operations, controlled entities may accumulate surpluses that are distributable via a dividend payment to the University. This policy sets out the University's requirements of its controlled entities relating to the payment of dividends to the University.
(2) This guideline applies to all controlled entities of the University whose corporate structure enables them to pay a dividend.
(3) This guideline is designed to confirm the requirement for controlled entities to provide a maximum return on capital invested to the University in the form of a cash dividend. This policy also recognises the need for controlled entities to maintain sufficient cash for operational purposes and, where appropriate, future development and growth.
(4) Prior to the end of each financial year a controlled entity shall declare a dividend to the University.
(5) Such dividend declared shall be paid to the University within 30 days of the end of the financial year.
the expected balance of Retained profits at the end of the financial year.
A description of the reasons for a reduced dividend amount supported by operational and capital expenditure plans.
Relevant supporting financial information to enable the University to undertake an assessment of the entity's financial performance, position and cash flow requirements.


text:
Why it can be hard to diagnose fibromyalgia
On behalf of Warren & Siurek, L.L.P. on Monday, July 1, 2019.
If you don't suffer from fibromyalgia, you might not understand the condition. Since it's largely invisible and there's no single, known source, some people may question if it even exists. But if you suffer from fibromyalgia, you know the pain and fatigue are real.
You might also find yourself dealing with a bad faith denial of your disability benefits. Insurance companies are always looking for ways to excuse themselves from making payments. They often deny fibromyalgia claims because they think they can get away with challenging the diagnosis.
How do doctors diagnose fibromyalgia?
Although it's hard to diagnose fibromyalgia, it's not impossible. Doctors have long acknowledged the condition is real. However, there's no single cause. This means there's no single test. Instead, doctors need to track the symptoms, which may include:
Lasting, widespread pain that can appear in different parts of your body
Grogginess and fatigue that can make it difficult to think clearly
Sensitivity to light, noise and temperature
Problems with the urinary tract and bowels
The tricky part often comes when doctors check against the other possible causes. These include:
Rheumatic diseases, like rheumatoid arthritis and lupus, that start with aches and pains
Depression, anxiety and other mental health issues
Neurological conditions that can cause the numbness and tingling people with fibromyalgia often feel
Doctors often run a series of tests to rule out these other possible causes. And it gets even harder to diagnose the condition if you have another condition at the same time.
Your ERISA protections
Thanks to the Employee Retirement Income Security Act (ERISA), your employer needs to give you a clear summary of your healthcare benefits. And you have the right to fight against disability denials made in bad faith. Too often, victims of fibromyalgia face these unfair denials.
Fortunately, researchers at The University of Texas may have made a recent breakthrough. They found links between fibromyalgia and insulin resistance. If it's confirmed, their research may someday make it easier to diagnose the condition. But for now, getting a diagnosis is still largely a process of elimination.
Americans with Disabilities Act (30)
Benefits & Compensation (5)
ERISA (20)
Employment Disputes (120)
Pension, Benefits & Compensation (7)
Wage & Hour Laws (61)
wage and hour laws (33)
workplace discrimination (71)
An overview of equal pay laws
Soccer becomes a game of hiding the dollar
New law in Texas secures minimum wage for those with disabilities
© 2019 by Warren & Siurek, L.L.P.. All rights reserved. Disclaimer

redpajama_set_name: RedPajamaCommonCrawl


Blandine is an executive-level global marketing professional and strategist who specialises in infusing market and consumer insights into marketing strategies that generate brand equity, awareness and revenue. Her previous experience includes working for Diego, Heineken and ESPN where she drove brand equity and volume growth, created profitable marketing partnerships with consumer brands and has managed teams of more than 16-persons both direct and cross-functionally, external agencies and marketing budgets of up to $30 million per year. She is currently at Hasbro as the Senior Director, Global Brand Strategy and Marketing where she is building brand strategies, and product and marketing plans for the brand.
After her presentation at ForwardLeading's Digital Marketing Leaders Summit in Boston, we talk to Blandine about her journey in marketing so far, her thoughts on data and digital marketing today and her predictions for the industry's future.
1) What have the key milestones in your brand's digital marketing journey been?
One of the biggest milestones was what one of my brand's, Red Stripe, experienced in the last 12 months - the development of a solid global social media platform. It sounds simple but when you are trying to make a brand global, the nuance of trying to make your brand relevant to consumers on different continents, languages and cultures is more challenging than it looks. Keeping a brand's global message uniform from a global perspective and allowing markets to customize their conversations with consumers in a distinct way is important and it also needs to managed closely.
2) What are the biggest mistakes you see marketers making with executing a digital-first strategy?
Saying digital-first strategy is the biggest mistake because really the conversation should be centered around being mobile-first. Consumers are more mobile than ever and figuring out how to serve content in this new environment was what marketers need to focus on. It's not about serving ads to consumers on the go via their mobile devices. It should be about creating immersive experiences that can be accessed on any device at any time.
3) How has the concept of 'data-driven marketing' evolved in your team?
Data-driven marketing has evolved to social-insights driven marketing; brands are trying to use social media listening to stay relevant in their category and to open up new categories of growth. Now, when we start discussing new ideas and strategies, we start the conversation with what is trending on social to see if there are insights there that can help guide our marketing strategy.
4) Are mobile and video the way forward for retail brands?
Yes and yes! I believe retail brands need to pay attention to what is happening with Amazon Go. This concept of checkout-free stores is really going to disrupt the retail space and once consumers adapt to the convenience of it which they certainly will, it will be hard to contain this disruption. What Amazon is offering with this proposition is a way to engage, deliver and serve consumers in one unique ecosystem. Brands will need to find ways to make the experience with their brands more intrusive and inclusive and it must be a balanced approach.
5) What are the new skills that you have had to develop to do your job effectively in the last few years?
I am developing my ability to identify social trends that matter to my portfolio has become more important than reading a Kantar or TNS report. It's incredible when you see and realize how much impact social media has on brands in both positive and negative ways. We can all cite numerous brands that are winning and losing because of social media.
ForwardLeading will be bringing the next Digital Marketing Leaders Summit to Singapore on 12 - 13 September 2018, and will return to Boston 20 - 21 June 2019.


text:
go-matsuoka
===========

Gou Matsuoka is an anime character from Free!, she's the
manager of the swimming club.

I'm a programmer, turned manager who still loves to build
code. I was getting tired of doing resource management on
spreadsheets; so I decided to start writing this.

The goal is to build a command-line resource manager.

Installation
------------

Currently there is no gem for this, because it is still
in its early stages. To install:

- Clone the repository
- Do a bundle install
- Run: ruby -Ilib bin/go-matsuoka.rb

Contributing
------------

Help us improve this:

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

TODO:
------------

- Write tests for resource, and resource-type models
- Write the functions to call when in the IRB
- Ability to seed data

redpajama_set_name: RedPajamaGithub