text:
\section{Introduction}

For an autonomous vehicle, understanding the dynamics of the surrounding environment is critical to ensure safe planning and navigation. It is essential for a self-driving vehicle to be able to perceive the actions of various entities around it, such as other vehicles, pedestrians, and cyclists. In the context of data recorded as 3D point clouds, a motion can be estimated for each 3D point; this is known as scene flow, which refers to the 3D velocity of each 3D point in a scene.
Its 2D analog, optical flow, is the projection of scene flow onto the image plane of a camera. An alternative to scene flow estimation is to use 3D object detection for object-level tracking and to assume that all points within a bounding box have the same rigid motion. However, in such a pipeline, errors in object detection can lead to errors in tracking. In contrast, scene flow methods can avoid such errors by directly estimating the motion of each 3D point.

Recent state-of-the-art methods learn to estimate the scene flow from 3D point clouds~\cite{liu2019flownet3d,gu2019hplflownet,wang2018deep, pointpwcnet}. However, these methods are fully supervised and require annotated datasets for training.  Such annotations are costly to obtain as they require labeling the motion for every point in a scene. To compensate for the lack of real world data, learning-based methods for scene flow have been trained primarily on synthetic datasets and fine tuned on real world data. This requirement of labeled training data limits the effectiveness of such methods in real world settings.

\begin{figure}[t]
 	\centering
 	\includegraphics[trim={5 10 25 0}, clip,width=0.45\textwidth]{figures/overview_car.pdf}
 	\caption{We use two self-supervised losses to learn scene flow on large unlabeled datasets.  The ``nearest neighbor loss" penalizes the distance between the predicted point cloud (\textcolor{green}{green}) and each predicted point's nearest neighbor in the second point cloud (\textcolor{red}{red}).  To avoid degenerate solutions, we also estimate the flow between these predicted points (\textcolor{green}{green}) in the reverse direction back to the original point cloud (\textcolor{blue}{blue}) to form a cycle. The new predicted points from the cycle (\textcolor{magenta}{purple}) should align with the original points (\textcolor{blue}{blue}); the distance between these two set of points forms our second self-supervised loss: ``cycle consistency."}
 	\label{fig:cycle}
  	\vspace{-1em}
\end{figure}

To overcome this limitation, we propose a self-supervised method for scene flow estimation. Using a combination of two self-supervised losses, we are able to mimic the supervision created by human annotation. Specifically, we use a cycle consistency loss, which  ensures that the scene flow produced is consistent in time (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot we ensure that a temporal cycle ends where it started). We also use a nearest neighbor loss; due to the unavailability of scene flow annotations, we consider the nearest point to the predicted translated point, in the temporally next point cloud, as the pseudo-ground truth association.  Intuitively, the nearest neighbor loss pushes one point cloud to flow toward occupied regions of the next point cloud. We show that this combination of losses can be used to train a scene flow network over large-scale, unannotated datasets containing sequential point cloud data. An overview of our method can be found in Figure~\ref{fig:cycle}.

We test our self-supervised training approach using the neural network architecture of a state-of-the-art scene flow method~\cite{liu2019flownet3d}.  The self-supervision allows us to train this network on large-scale, unlabeled autonomous driving datasets. Our method matches the current state-of-the-art performance when no real world annotations are given. Moreover, our method exceeds the performance of state-of-the-art scene flow estimation methods when combined with supervised learning on a smaller labeled dataset.

\section{Related Work}
\paragraph*{Scene Flow} Vedula \etal~\cite{vedula1999three} first introduced the task of scene flow estimation. They propose a linear algorithm to compute it from optical flow. Other works involve joint optimization of camera extrinsics and depth estimates for stereo scene flow~\cite{valgaerts2010joint}, use of particle filters~\cite{hadfield2011kinecting}, local rigid motion priors~\cite{vogel20113d,vogel20153d,vogel2013piecewise, menze2015object}, and smoothness-based regularization~\cite{basha2013multi}.

\vspace{-1em}
\paragraph*{Deep Scene Flow}
State-of-the-art scene flow methods today use deep learning to improve performance.
FlowNet3D~\cite{liu2019flownet3d} builds on PointNet++~\cite{qi2017pointnet++, qi2017pointnet} to estimate scene flow directly from a pair of point clouds.
Gu \etal ~\cite{gu2019hplflownet} produced similar results using a permutohedral lattice to encode the point cloud data in a sparse, structured manner.  The above approaches compute scene flow directly from 3D point clouds~\cite{liu2019flownet3d,gu2019hplflownet}. Methods involving voxelizations with object-centeric rigid body assumptions~\cite{Behl_2019_CVPR}, range images~\cite{vaquero2018deep}, and non-grid structured data~\cite{wang2018deep} have also been used for scene flow estimation. All of the above methods were trained either with synthetic data~\cite{mayer2016large} or with a small amount of annotated real-world data~\cite{kitti} (or both).  Our self-supervised losses enable training on large unlabeled datasets, leading to large improvements in performance.

\vspace{-1em}
\paragraph*{Self-Supervised Learning} Wang \etal ~\cite{wang2019learning} used self-supervised learning for 2D tracking on video. They propose a tracker which takes a patch of an image at time $t$ and the entire image at time $t-1$ to track the image patch in the previous frame. They define a self-supervised loss by tracking the patch forward and backward in time to form a cycle while penalizing the errors through cycle consistency and feature similarity. We take inspiration from this work for our self-supervised flow estimation on point clouds. Other works including self-supervisory signals are image frame ordering~\cite{misra2016shuffle}, feature similarity over time~\cite{wang2015unsupervised} for images, and clustering and reconstruction~\cite{hassani2019ICCV} from point clouds. While these can potentially be used for representation learning from 3D data, they cannot be directly used for scene flow estimation. Concurrent to our work, Wu \etal~\cite{pointpwcnet} showed that Chamfer distance, smoothness constraints, and Laplacian regularization can be used to train scene flow in a self-supervised manner.

\section{Method}

\subsection{Problem Definition}
For the task of scene flow estimation, we have a temporal sequence of point clouds: point cloud $\mathcal{X}$ as the point cloud captured at time $t$ and point cloud $\mathcal{Y}$ captured at time $t+1$. There is no structure enforced on these point clouds and they can be recorded directly from a LIDAR sensor or estimated through a stereo algorithm.
Each point $p_i = \{x_i,f_i\}$ in point cloud $\mathcal{X}$ contains the Cartesian position of the point, $x_i \in \mathbb{R}^3$, as well as any additional information which the sensor produces, such as color, intensity, normals, etc, represented by $f_i \in \mathbb{R}^c$.

The scene flow, $\mathcal{D} = \{d_i\}^N, d_i \in \mathbb{R}^3$ between these two point clouds describes the movement of each point $x_i$ in point cloud $\mathcal{X}$ to its corresponding position $x_i'$ in the scene described by point cloud $\mathcal{Y}$, such that $x_i' = x_i + d_i$,  and $N$ is the size of point cloud $\mathcal{X}$. Scene flow is defined such that $x_i$ and $x'_i$ represent the same 3D point of an object moved in time. Unlike optical flow estimation, the exact 3D position of $x_i'$ may not necessarily coincide with a point in the point cloud $\mathcal{Y}$, due to the sparsity of the point cloud. Additionally, the sizes of $\mathcal{X}$ and $\mathcal{Y}$ may be different.

\vspace{-1em}
\paragraph*{Supervised Loss}
The true error associated with our task is the difference between the estimated flow $g(\mathcal{X}, \mathcal{Y}) = \mathcal{\hat{D}} = \{\hat{d}_i\}^N$ and the ground truth flow $\mathcal{D}^* = \{d^*_i\}^N$,
\begin{align}
\mathcal{L}_{gt} = \frac{1}{N} \sum_i^N \|d_i^*-\hat{d}_i\|^2.
\label{equ:gt}
\end{align}
The loss in Equation~\ref{equ:gt} is useful because it is mathematically equivalent to the end point error, which we use as our evaluation metric. However, computing this loss requires annotated ground truth flow $d^*_i$. This type of annotation is easy to calculate in synthetic data~\cite{mayer2016large}, but requires expensive human labeling for real world datasets. As such, only a small amount of annotated scene flow datasets are available ~\cite{menze2015joint, menze2018object}. While training on  purely synthetic data is possible, large improvements can often be obtained by training on real data from the domain of the target application.  For example, Lui \etal~\cite{liu2019flownet3d} showed an 18\% relative improvement after fine-tuning on a small amount of annotated real world data.  This result motivates our work to use self-supervised training to train on large unlabeled datasets.

\begin{figure}[t]
 	\centering
 	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={70 250 70 250}, clip, width=\linewidth]{figures/car_loss_forward_30.pdf}

 	\caption{Nearest Neighbor Loss}
 	\end{subfigure}

  	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={70 250 70 250}, clip, width=\linewidth]{figures/car_loss_cycle_30.pdf}
 	\caption{Cycle Consistency Loss}
 	\end{subfigure}
 	\caption{Example of our self-supervised losses between consecutive point clouds $\mathcal{X}$ (\textcolor{blue}{blue}) and $\mathcal{Y}$ (\textcolor{red}{red}). We consider the point $x$ whose ground truth projected point $x'$ is not known during training time. $(a)$ Nearest Neighbor Loss is computed between the projected point $\hat{x}'$, predicted by the forward flow, and the closest point in $\mathcal{Y}$.  $(b)$ The Cycle Consistency Loss tracks this transformed point back onto its original frame, as point $\hat{x}''$, using the reverse flow, and computes the distance to its original position ${x}$.}
 	\label{fig:losses}
  	\vspace{-1em}
\end{figure}

\vspace{-1em}
\paragraph*{Nearest Neighbor (NN) Loss}

For large unlabeled datasets, since we do not have information about $d_i^*$, we cannot compute the loss in Equation~\ref{equ:gt}.
In lieu of annotated data, we take inspiration from methods such as Iterative Closest Point~\cite{besl1992method} and use the nearest neighbor of our transformed point $\hat{x}_i' = x_i + \hat{d}_i$ as an approximation for the true correspondence. For each transformed point $\hat{x}_i' \in \hat{\mathcal{X}}'$, we find its nearest neighbor $y_j \in \mathcal{Y}$ and compute the Euclidean distance to that point, illustrated as $e_{NN}$ in Figure~\ref{fig:losses}a:
\begin{equation}
    \mathcal{L}_{NN} = \frac{1}{N}\sum_i^N \min_{y_j \in \mathcal{Y}}\|\hat{x}_i' - y_j\|^2.
\label{equ:nn}
\end{equation}

Assuming that the initial flow estimate is sufficiently close to the correct flow estimate, this loss will bring the transformed point cloud and the target point cloud closer. This loss can, however, have a few drawbacks if imposed alone.  First, the true position of the point $x_i$ transformed by the ground truth flow, $x_i' = x_i + d_i^*$, may not be the same as the position of the nearest neighbor to $\hat{x}'$ (transformed by the \emph{estimated} flow) due to potentially large errors in the estimated flow, as illustrated in Figure~\ref{fig:losses}a.  Further, the position of $x_i'$ may not correspond with any point in $\mathcal{Y}$ if the point cloud $\mathcal{Y}$ is sufficiently sparse, as is common for point clouds collected by sparse 3D LIDAR for autonomous driving.  Finally, this loss does not penalize degenerate solutions where all of the points in $\mathcal{X}$ map to the same point in $\mathcal{Y}$; such a degenerate solution would obtain 0 loss under Equation~\ref{equ:nn}. To address these issues, we use an additional self-supervised loss: cycle consistency loss.

\vspace{-1em}
\paragraph*{Cycle Consistency Loss}
To avoid the above issues, we incorporate an additional self-supervised loss: cycle consistency loss, illustrated in Figure~\ref{fig:losses}b. We first estimate the ``forward" flow as $\hat{\mathcal{D}} = g(\mathcal{X}, \mathcal{Y})$.  Applying the estimated flow $\hat{d}_i \in \hat{\mathcal{D}}$ to each point $x_i \in \mathcal{X}$ gives an estimate of the location of the point $x_i$ in the next frame: $\hat{x}_i' = x_i + \hat{d}_i$.  We then compute the scene flow in the reverse direction: for each transformed point $\hat{x}_i'$ we estimate the flow to transform the point back to the original frame, $\hat{\mathcal{D'}} = g(\mathcal{\hat{X}'}, \mathcal{X})$.  Transforming each point $\hat{x}_i'$ by this ``reverse" flow $\hat{d}_i'$ gives a new estimated point $\hat{x}_i''$.  If both the forward and reverse flow are accurate, this point $\hat{x}_i''$ should be the same as the original point $x_i$.  The error between these points, $e_{cycle}$, is the ``cycle consistency loss,"  given by
\begin{equation}
    \mathcal{L}_{cycle} = \sum_i^N \|\hat{x}_i'' - x_i\|^2.
\label{equ:cycle}
\end{equation}
A similar loss is used as a regularization in~\cite{liu2019flownet3d}.

However, we found that implementing the cycle loss in this way can produce unstable results if only self-supervised learning is used without any ground-truth annotations.  These instabilities appear to be caused by errors in the estimated flow which lead to structural distortions in the transformed point cloud $\hat{\mathcal{X}}'$, which is used as the input for computing the reverse flow $g(\mathcal{\hat{X}'}, \mathcal{X})$. This requires the network to simultaneously learn to correct any distortions in $\mathcal{\hat{X}'}$, while also learning to estimate the true reverse flow.
To solve this problem, we use the nearest neighbor $y_j$ of the transformed point $\hat{x}'_i$ as an anchoring point in the reverse pass.  Using the nearest neighbor $y_j$ stabilizes the structure of the transformed cloud while still maintaining the correspondence around the cycle. The effects of this stabilization are illustrated in Figure~\ref{fig:anchor}. As we are using the anchoring point as part of the reverse pass of the cycle, we refer to this loss as ``anchored cycle consistency loss''.

Specifically, we compute the anchored reverse flow as follows. First, we compute the forward flow as before, $\hat{\mathcal{D}} = g(\mathcal{X}, \mathcal{Y})$, which we use to compute the transformed point cloud $\hat{x}_i' = x_i + \hat{d}_i$.  We then compute anchor points $\bar{\mathcal{X}'} = \{\bar{x}_i'\}^N$ as a convex combination of the transformed point and its nearest neighbor $\bar{x}_i' = \lambda \hat{x}_i' + (1-\lambda) y_j$.  In our experiments, we find that $\lambda=0.5$ produces the most accurate results. Finally, we compute the reverse flow using these anchored points: $\bar{\mathcal{D}}' = g(\bar{\mathcal{X}}', \mathcal{X})$.  The cycle loss of Equation~\ref{equ:cycle} is then applied to this anchored reverse flow.  By using anchoring, some of the structural distortion of the predicted point cloud $\mathcal{\hat{X}'}$ will be removed in the anchored point cloud $\bar{\mathcal{X}'}$, leading to a more stable training input for the reverse flow.

Note that the cycle consistency loss also has a degenerate solution: the ``zero flow,", \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot $\hat{\mathcal{D}} = 0$, will produce 0 loss according to Equation~\ref{equ:cycle}.  However, the zero flow will produce produce a non-zero loss when anchored cycle consistency is used; thus anchoring helps to remove this degenerate solution.  Further, the nearest neighbor loss will also be non-zero for the degenerate solution of zero flow.  Thus, the local minima of each of the nearest neighbor and cycle consistency losses conflict, allowing their sum, $\mathcal{L} = \mathcal{L}_{NN} + \mathcal{L}_{cycle}$, to act as a stable surrogate for the true error.

\begin{figure}[t]
 	\centering
 	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={20 70 20 70}, clip, width=\linewidth]{figures/car_noise.png}
 	\caption{}
 	\end{subfigure}
 	\hspace*{\fill}
  	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={20 70 20 70}, clip, width=\linewidth]{figures/car_anchor.png}
 	\caption{}
 	\end{subfigure}
 	\caption{Compounding errors cause problems in estimating reverse flow using the transformed point cloud. $(a)$ Large flow prediction errors degrade the structure of the transformed cloud $\hat{\mathcal{X}}'$ (shown in \textcolor{green}{green}). Thus, computing the reverse flow between $\hat{\mathcal{X}}'$ (\textcolor{green}{green}) and $\mathcal{X}$ (\textcolor{blue}{blue}) is an ill-posed task.
 	$(b)$ Using the nearest neighbor points (\textcolor{red}{red}) as anchors, we are able to stabilize the transformed cloud $\bar{\mathcal{X}}'$ (\textcolor{cyan}{cyan}), thus retaining important structural information.}
 	\label{fig:anchor}
  	\vspace{-1em}
\end{figure}

\vspace{-1em}
\paragraph*{Temporal Flip Augmentation}

Having a dataset of point cloud sequences in only one direction may generate a motion bias which may lead to the network predicting the flow equal to the average forward speed of the training set. To reduce this bias, we augment the training set by flipping the point clouds, i.e, reversing the flow. With this augmentation, the network sees an equal number of point cloud sequences having a forward motion and a backward motion.

\section{Experiments}

We run several experiments to validate our self-supervised method for scene flow estimation for various levels of supervision and different amounts of data. First, we show that our method, with self-supervised training on large unlabeled datasets, can perform as well as supervised training on the existing labeled data.
Next, we investigate how our results can be improved by combining self-supervised learning with a small amount of supervised learning, exceeding the performance of purely supervised learning. Finally, we explore the utility of each element of our method through an ablation study.

\subsection{Implementation Details}
For all data configurations (our method and the baseline), we initialize our network with the parameters of the Flownet3D model~\cite{liu2019flownet3d} pre-trained on the FlyingThing3D dataset~\cite{mayer2016large}. We compare our self-supervised training procedure to a baseline which uses supervised fine-tuning on the KITTI dataset~\cite{kitti}.  The baseline used in the comparison is the same as in Liu \etal~\cite{liu2019flownet3d}, except that we increase the number of training iterations from 150 epochs (as described in the original paper) to 10k epochs in order to keep the number of training iterations consistent with that used in our self-supervised method. We see that this change leads to a small improvement in the baseline performance, which we include in the results table.

\begin{figure*}[t]
 	\centering
 	\includegraphics[trim={0 0 0 50}, clip, width=\textwidth]{figures/self-supervised-unsupervised-f3d.pdf}
 	\caption{Scene flow estimation between point clouds at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green}) from the KITTI dataset trained without any labeled LIDAR data. Predictions from our self-supervised method, trained on nuScenes and fine-tuned on KITTI using self-supervised learning is shown in \textcolor{blue}{blue}; the baseline with only synthetic training is shown in \textcolor{magenta}{purple}. In the absence of real-world supervised training, our method clearly outperforms the baseline method, which overestimate the flow in many regions. (Best viewed in color)}
 	\label{fig:qual_no_ground_truth}
\end{figure*}

\begin{figure*}[t]
 	\centering

 	\begin{subfigure}{0.48\textwidth}
 	\includegraphics[trim={0 0 900 50}, clip, width=\textwidth]{figures/nuscenes_comparison.pdf}
 	\caption{Ours (Self-Supervised Fine Tuning)}
 	\end{subfigure}

  	\begin{subfigure}{0.48\textwidth}
 	\includegraphics[trim={920 0 0 50}, clip, width=\textwidth]{figures/nuscenes_comparison.pdf}
 	\caption{Baseline (No Fine Tuning)}
 	\end{subfigure}
 	\caption{Comparison of our self-supervised method to a baseline trained only on synthetic data, shown on the nuScenes dataset. Scene flow is computed  between point clouds at time $t$~(\textcolor{red}{red}) and $t+1$~(\textcolor{green}{green}); the point cloud that is transformed using the estimated flow is in shown in \textcolor{blue}{blue}. In our method, the predicted point cloud has a much better overlap with the point cloud of the next timestamp (\textcolor{green}{green})  compared to the baseline. Since nuScenes dataset does not provide any scene flow annotation, the supervised approaches cannot be fined tuned to this environment.}
 	\label{fig:qual_nuscenes}
 	\vspace{-1em}
\end{figure*}

\begin{figure*}[ht!]
 	\centering
 	\includegraphics[trim={0 0 0 40}, clip, width=\textwidth]{figures/supervised_new.pdf}
 	\caption{Scene flow estimation on LIDAR data from the KITTI dataset between point clouds at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green}). Our method, which is trained on nuScenes using self-supervised learning and then fine-tuned on KITTI using supervised learning, is shown in \textcolor{blue}{blue}. The baseline method is fine-tuned only on KITTI using supervised training and is shown in \textcolor{magenta}{purple}. While in aggregate, both methods well estimate the scene flow, adding self-supervised training on nuScenes (\textcolor{blue}{blue}) enables our predictions to more closely match the next frame point cloud (\textcolor{green}{green}). In several  scenes, the purely supervised method (\textcolor{magenta}{purple}) underestimates the flow, staying too close to the initial point cloud (\textcolor{red}{red}). (Best viewed in color)}
 	\label{fig:qual_supervised}

\end{figure*}

\subsection{Datasets}

\paragraph*{KITTI Vision Benchmark Suite}  KITTI~\cite{kitti} is a real-world self-driving dataset. There are 150 scenes of LIDAR data in KITTI collected using seven scans of a Velodyne 64 LIDAR, augmented using 3D models, and annotated with ground truth scene flow~\cite{menze2018object}. For our experiments under both self-supervised and supervised settings, we consider 100 out of 150 scenes for training and the remaining 50 scenes for testing. Ground points are removed from every scene using the pre-processing that was performed in previous work~ \cite{liu2019flownet3d}. Every scene consists of a pair of point clouds recorded at two different times as well as the ground truth scene flow for every point of the first point cloud.

\vspace{-1em}
\paragraph*{nuScenes} The nuScenes~\cite{nuscenes2019} dataset is a large-scale public dataset for autonomous driving. It consists of 850 publicly available driving scenes in total from Boston and Singapore. The LIDAR data was collected using a Velodyne 32 LIDAR rotating at 20 Hz. This is in contrast to the 64-beam Velodyne rotating at 10 Hz used for the KITTI dataset.  This difference in sensors leads to a difference in data sparsity that creates a distribution shift between KITTI and nuScenes.  This distribution shift necessitates additional training on KITTI beyond our self-supervised training on nuScenes.  Nonetheless, our results show a substantial benefit from the self-supervised training on nuScenes.

Since the nuScenes dataset~\cite{nuscenes2019} does not contain scene flow annotations, we must use self-supervised methods when working with this dataset. In our experiments, out of the 850 scenes available, we use 700 as the train set and the rest 150 as the validation set. Similar to KITTI, we remove the ground points from each point cloud using a manually tuned height threshold.

\begin{table*}[h]
\centering
\begin{tabular} {l|c c c}
\toprule
 Training Method & EPE (m) $\downarrow$ & ACC (0.05) $\uparrow$ & ACC (0.1) $\uparrow$ \\
\hline
\hline
No Fine Tuning & 0.122 & 25.37\% & 57.85\% \\
KITTI (Supervised) &
0.100 & 31.42\% & 66.12\% \\
\hline
Ablation: KITTI (Self-Supervised)  & 0.126 & 32.00\% & 73.64\% \\
Ours: nuScenes (Self-Supervised) + KITTI (Self-Supervised)  & 0.105 & 46.48\% & 79.42\% \\
Ours: nuScenes (Self-Supervised) + KITTI (Supervised) & \textbf{0.091} & \textbf{47.92\%} & \textbf{79.63\%} \\
\bottomrule
\end{tabular}
\caption{Comparison of levels of supervision on KITTI dataset. The nearest neighbor + anchored cycle loss is used for nuScenes (self-supervised) and KITTI (self-supervised). All methods are pretrained on FlyingThings3D\cite{mayer2016large} and ground points are removed for KITTI and nuScenes datasets.}\label{tbl:supervision}
\vspace{-1em}
\end{table*}

\subsection{Results}
We use three standard metrics to quantitatively evaluate the predicted scene flow when the ground truth annotations of scene flow are available. Our primary evaluation metric is End Point Error (EPE) which describes the mean Euclidean distance between the predicted and ground truth transformed points, described by Equation~\ref{equ:gt}. We also compute accuracy at two threshold levels, Acc(0.05) as the percentage of scene flow prediction with an EPE~$<~0.05$m or relative error~$<5\%$ and Acc(0.1) as percentage of points having an EPE~$<0.1$m or relative error~$<~10\%$, as was done for evaluation in previous work~\cite{liu2019flownet3d}.

\subsubsection{Quantitative Results}

\paragraph*{Self-supervised training:}
Unlike previous work, we are not restricted to annotated point cloud datasets; our method can be trained on any sequential point cloud dataset. There are many point cloud datasets containing real LIDAR captures of urban scenes, but most of them do not contain scene flow annotations. Due to lack of annotations, these datasets can not be utilized for supervised scene flow learning. In contrast, our self-supervised loss allows us to easily integrate them into our training set. The combination of these datasets (KITTI + NuScenes) contains 5x more real data than using KITTI alone.

The results are shown in Table~\ref{tbl:supervision}.
To show the value of self-supervised training, we evaluate the performance of our method without using any  ground-truth annotations.  We first pre-train on the synthetic FlyingThings3D dataset; we then perform self-supervised fine-tuning on the large nuScenes dataset followed by further self-supervised fine-tuning on the smaller KITTI dataset (4th row: ``Ours: nuScenes (Self-Supervised) + KITTI (Self-Supervised)").  As can be seen,
using no real-world annotations, we are able to achieve an EPE of 0.105~m.  This outperforms the baseline of only training on synthetic data (``No Fine Tuning").  Even more impressively, our approach performs similarly to the baseline which
pre-trains on synthetic data and then
does supervised fine-tuning  on the KITTI dataset (``KITTI (Supervised)"); our method has a similar EPE and outperforms this baseline in terms of accuracy, despite not having access to any annotated training data. This result shows that our method for self-supervised training, with a large enough unlabeled dataset, can match the performance of supervised training.

\vspace{-1em}
\paragraph*{Self-supervised + Supervised:} Finally, we show the value of combining our self-supervised learning method with a small amount of supervised learning. For this analysis, we perform self-supervised training on NuScenes as above, followed by supervised training on the much smaller KITTI dataset.  The results are shown in the last row of Table~\ref{tbl:supervision}.

As can be seen, this approach of self-supervised training followed by supervised fine-tuning outperforms all other methods on this benchmark, obtaining an EPE of 0.091, outperforming the previous state-of-the-art result which used only supervised training.  This shows the benefit of self-supervised training on large unlabeled datasets to improve scene flow accuracy, even when scene flow annotations are available.

While Table~\ref{tbl:supervision} only shows results using the FlowNet3D~\cite{liu2019flownet3d} architecture, we note that our method also outperforms the results of HPLFlownet~\cite{gu2019hplflownet}~(EPE of 0.1169) and all models they compare against as well.

Figure \ref{fig:epe_vs_mag} provides an analysis on the correlation between average endpoint error and the magnitude of the ground truth flow.  As can be seen, our method consistently outperforms the baseline at almost all flow magnitudes.

\begin{figure}[t]
 	\centering
 	\includegraphics[width=0.47\textwidth]{figures/self-super-vs-unsuper-new.png}
 	\caption{Analysis of average EPE (m) with respect to ground truth flow magnitudes (m). Flow estimates are binned by ground truth flow magnitude and a confidence interval of 95\% is shown for all results.}
 	\label{fig:epe_vs_mag}
 	\vspace{-1em}
\end{figure}

\subsubsection{Qualitative Analysis}
\paragraph*{Self-supervised training - KITTI results:}
Next we perform a qualitative analysis to visualize the performance of our method.
We compare our method (synthetic pre-training + self-supervised training on nuScenes + self-supervised training on KITTI) compared to the baseline of synthetic training only.  Results on KITTI are shown in Figure~\ref{fig:qual_no_ground_truth}. The figure shows the point clouds captured at time $t$ and $t+1$  in red and green, respectively. The predictions from our method are shown in blue and the baseline predictions are shown in purple. As shown, our scene flow predictions (blue)  have a large overlap with the point cloud at time $t+1$ (green). On the other hand, the baseline predictions (purple) do not overlap with the point cloud at time $t+1$.  The baseline, trained only on synthetic data, fails to generalize to the real-world KITTI dataset. On the contrary, our self-supervised approach can be fine tuned on any real world environment and shows a significant improvement over the baseline.

\vspace{-1em}
\paragraph*{Self-supervised training - nuScenes results:}

Next, we visualize the performance of our method on the nuScenes dataset.  Note that, because nuScenes does not have scene flow annotations, only qualitative results can be shown on this dataset. For this analysis, our method is pre-trained on synthetic data (FlyingThings3D) as before and then fine-tuned on nuScenes in a self-supervised manner. No scene flow annotations for nuScenes are available, so we compare to a baseline which is trained only on synthetic data.

The results on nuScenes are shown in Figure~\ref{fig:qual_nuscenes}. These results again showcase the advantages of self-supervision on real world data over purely synthetic supervised training. As before, the figure shows the point clouds captured at time $t$ and $t+1$ in red and green respectively. The predictions are shown in blue, with our method on the left (Fig.~\ref{fig:qual_nuscenes}a) and the baseline on the right (Fig.~\ref{fig:qual_nuscenes}b).  As shown, our scene flow predictions (left figure, blue) have a large overlap with the point cloud at time $t+1$ (green). On the other hand, the baseline predictions (right figure, blue) do not overlap with the point cloud at time $t+1$.

The low performance of the baseline can again be attributed to its training on synthetic data and its inability to generalize to real-world data.  For the nuScenes data, no scene flow annotations exist, so only self-supervised learning is feasible to improve performance.

\begin{table}[]
\centering
\footnotesize
\setlength\tabcolsep{2pt}
\begin{tabular}{c|c|c|c||c|c|c}
\toprule
\multicolumn{7}{l}{Self-Supervised Training (nuScenes + KITTI)} \\ \midrule
NN Loss & Cycle Loss & Anchor Pts & Flip & EPE (m) & ACC (0.05) & ACC (0.1) \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & 0.105 & 46.48\% & 79.42\% \\
\checkmark & \checkmark & \checkmark &  & 0.107 & 40.03\% & 72.20\% \\
\checkmark & \checkmark &  &  & 0.146 & 30.21\% & 48.57\% \\
\checkmark &  &  &  & 0.108 & 42.00\% & 78.51\% \\
 &  &  &  & 0.122 & 25.37\% & 57.85\% \\
\hline \hline
\multicolumn{7}{l}{Self-Supervised (nuScenes) + Supervised Training (KITTI)} \\ \hline
NN Loss & Cycle Loss & Anchor Pts & Flip & EPE (m) & ACC (0.05) & ACC (0.1) \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & {0.091} & {47.92\%} & {79.63\%} \\
\checkmark & \checkmark & \checkmark &  & {0.093} & {40.69\%} & {74.50\%} \\
\checkmark & \checkmark &  &  & 0.092 & 30.76\% & 72.94\% \\
\checkmark &  &  &  & 0.114 & 31.24\% & 64.58\% \\
 &  &  &  & 0.100 & 31.42\% & 66.12\%\\ \bottomrule
\end{tabular}
\caption{Ablation analysis: We study the effect of the different self-supervised losses and data augmentation. Top: Models use self-supervised training on nuScenes and KITTI; Bottom: Models use self-supervised training on nuScenes followed by supervised training on KITTI.}\label{tbl:ablation}
\vspace{-1em}
\end{table}

\begin{table}
\centering
\footnotesize
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c | c | c | c}
\toprule
$\lambda$ & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
0   & 0.120 & 24.09\% & 73.20\% \\
0.25 & 0.122 &  26.41\% & 65.57\% \\
0.5 & \textbf{0.105} & \textbf{46.48\%} & \textbf{79.42\%} \\
0.75 &  0.125 & 23.59\% & 62.96\% \\
1   & 0.149 & 22.97\% & 49.58\% \\
\bottomrule
\end{tabular}
\caption{Effect of varying the $\lambda$ parameter for ``anchoring" the Cycle Consistency Loss. Results are shown for self-supervised training on nuScenes + KITTI.}\label{tbl:cycle_lambda}
\vspace{-1em}
\end{table}

\vspace{-1em}
\paragraph*{Self-supervised + supervised training - KITTI results:}
Finally, we show the value of combining our self-supervised learning method with a small amount of supervised learning, compared to only performing supervised learning.  For our method, we perform synthetic pre-training, followed by self-supervised fine-tuning on nuScenes, followed by supervised fine-tuning on the much smaller KITTI dataset. We compare to a baseline which only performs synthetic pre-training followed by supervised fine-tuning KITTI.

Qualitative results can be seen in Figure~\ref{fig:qual_supervised}. The figure shows the point clouds captured at time $t$ and $t+1$  in red and green, respectively. The predictions from our method are shown in blue and the baseline predictions are shown in purple. As shown, our scene flow predictions (blue)  have a large overlap with the point cloud at time $t+1$ (green), whereas the baseline predictions (purple) do not.

The baseline predicts a small motion, keeping the transformed cloud (purple) too close to the initial position (red). As discussed above, this bias towards small motion is likely due to the training of the baseline over a synthetic dataset, which affects the generalization of the baseline to real-world datasets where objects exhibit different types of motion than seen in simulation. By training over a significantly larger unlabeled dataset, our method is able to avoid overfitting and generalizes better to the scenes and flow patterns which were not present in the synthetic dataset.

\begin{figure}[t]
 	\centering
 	\includegraphics[trim={0 0 0 100}, clip, width=\columnwidth]{figures/cycle_vs_forward_comparison.pdf}
 	\caption{Ablation study comparing our self-supervised method with both nearest neighbor loss and anchored cycle consistency loss (\textcolor{blue}{blue}) compared to training only using the nearest neighbor loss (\textcolor{magenta}{purple}). Scene flow is computed between point clouds from the KITTI dataset at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green})}
 	\label{fig:nuscenes_ablation}
  	\vspace{-1em}
\end{figure}

\subsubsection{Ablation Studies}

We test the importance of each component of our method by running a series of ablation studies. Table~\ref{tbl:ablation} shows the effect of iteratively removing portions of our method in both the purely self-supervised (top) and self-supervised + supervised (bottom) training.
The benefits of anchored cycle consistency loss (compared to only using nearest neighbor loss) can be seen in  Table~\ref{tbl:ablation} (bottom) as well as in Figure~\ref{fig:nuscenes_ablation}.

The benefits of anchoring is apparent by the large drop in performance for self-supervised training (top) when anchoring is removed.
 Introducing the anchor point cloud as part of the backward flow greatly improves performance when only self-supervised training is used (Figure~\ref{fig:nuscenes_ablation}, top).
  This suggests that having anchored point clouds stabilizes the training of the cycle consistency loss. Additionally, we evaluate the sensitivity of our system to the selection of the anchoring parameter, $\lambda$. Table~\ref{tbl:cycle_lambda} shows that we obtain the best results with $\lambda=0.5$, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot using the average of the predicted and nearest points. Overall, these analyses show the benefits of each component of our method. Further ablation analysis can be found in the supplement.

\section{Conclusion}

In this work, we propose a self-supervised method for training scene flow algorithms using a combination of cycle consistency in time and nearest neighbor losses. Our purely self-supervised method is able to achieve a performance comparable to that of the supervised methods on the widely used KITTI self-driving dataset. We further show that when supervised training is augmented with self-supervision on a large-scale, unannotated dataset, the results exceed the current state-of-the-art performance. Our self-supervision method opens the door to fine-tuning on arbitrary datasets that lack scene flow annotations.

\subsection*{Acknowledgements}
This work was supported by the CMU Argo AI Center for Autonomous Vehicle Research and a NASA Space Technology Research Fellowship.
\section*{Supplementary}
\subsection{Hyperparameters}
A batch size of 8 is used for all datasets. All models are trained using the Adam optimizer with a learning rate of 1e-4, $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=1e-8$.

\subsection{Further Ablation Studies}
In this section, we extend our ablation study to further evaluate the importance of each component used in our method. Beginning from our full method, we remove a single component to see the change on the evaluation metrics.

The results can be seen in Table~\ref{tbl:ablation}.  Because  ``anchoring" is a modification of the cycle loss, when Cycle Consistency Loss is removed, anchoring must be removed too.  The results in Table~\ref{tbl:ablation} show that each component is important for the performance of our method, and removing any of the components results in a drop in performance. This can especially be seen in the accuracy performance metrics (ACC 0.05 and 0.1), which show large drops when any component is removed.

\begin{table}[h]
\centering
\footnotesize
\setlength\tabcolsep{2pt}
\begin{tabular}{c|c|c|c||c|c|c}
\toprule
\multicolumn{7}{l}{Self-Supervised Training (nuScenes + KITTI)} \\ \midrule
NN Loss & Cycle Loss & Anchor  & Flip & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
           & \checkmark & \checkmark & \checkmark & 0.1768 & 15.90 & 35.81 \\
\checkmark &            &            & \checkmark & 0.1102 & 30.80 & 73.27 \\
\checkmark & \checkmark &  & \checkmark & 0.1493 & 22.97 & 49.58 \\
\checkmark & \checkmark &  \checkmark &  & 0.1072 & 40.03 & 72.20 \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & \textbf{0.1053} & \textbf{46.48} & \textbf{79.42} \\
\hline \hline
\multicolumn{7}{l}{Self-supervised (nuScenes) + Supervised Training (KITTI)} \\ \hline
NN Loss & Cycle Loss & Anchor & Flip & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
          & \checkmark & \checkmark & \checkmark & {0.1572} & {18.50} & {52.80} \\
\checkmark &            &            & \checkmark & {0.1090} & {34.88} & {71.32} \\
\checkmark & \checkmark &  & \checkmark & {0.0932} & {28.18} & {66.10} \\
\checkmark & \checkmark &  \checkmark &  & 0.0926 & 40.69 & 74.50 \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & \textbf{0.0912} & \textbf{47.92} & \textbf{79.63}  \\
\bottomrule
\end{tabular}
\caption{Leave-one-out ablation analysis: We study the effect of removing a single component of self-supervised loss and data augmentation. Top: Models use self-supervised training on nuScenes and KITTI; Bottom: Models use self-supervised training on nuScenes followed by supervised training on KITTI.}\label{tbl:ablation}
\vspace{-1em}
\end{table}

\subsection{Analysis of Point Density vs End Point Error}

We analyze the correlation between local point density and endpoint error. For each point in the point cloud at time $t$, we compute the number of points within a 0.1m radius neighborhood of that point. Each point is binned based on the density of its local neighborhood. For each bin, the mean end-point error (EPE) is computed for both the baseline, trained only synthetic data, and our method, trained on nuScenes and KITTI using our self-supervised losses. Figure \ref{fig:query_ball} shows no correlation between EPE and neighborhood density for either the baseline or for our method.

\begin{figure}
  \includegraphics[width=\linewidth]{figures/plots_query_ball_new.png}
  \caption{Analysis of average EPE (m) with respect to the local point density (points/$m^3$). Flow estimates are binned by local point density and a confidence interval of 95\% is shown for all results.}
  \label{fig:query_ball}
\end{figure}

\subsection{Error Distribution of End Point Error}

We analyze the error distributions of our method and a baseline method. Our method uses self-supervised training on nuScenes followed by supervised training on KITTI. The baseline is trained only on KITTI using supervised learning. By computing the error at every point of every scan in the KITTI test set, we can view the full distribution of end point errors (EPE), shown in Figure~\ref{fig:error_distribution}. To better show the effects of outliers, we use log binning for the x-axis. Not only is the center of our error distribution lower, with an average EPE of 0.091, but it also shows fewer large outliers than the baseline method.

\begin{figure}
  \includegraphics[width=\linewidth]{figures/ours_sup_vs_base_sup_final.png}
  \caption{End point error distribution of our method (\textcolor{blue}{blue}) and the baseline (\textcolor{orange}{orange}).}
  \label{fig:error_distribution}
\end{figure}
\subsection{Implementation Details}

\paragraph*{Setup} We fine tune on KITTI dataset in a self-supervised manner using our model. Although KITTI provides ground truth scene flow annotations for the point clouds, we perform this experiment to test the generalization ability of our model. KITTI is a small-scale dataset and therefore to finetune on KITTI effectively, we initialize the network with the parameters of the model pre-trained on large-scale nuScenes dataset in a self-supervised manner as described above. For this set of experiments, we again train the model for 10K epochs with a batch size of 8. Since KITTI has scenes with two point clouds each, we simply pick the two point clouds as input to the model in each iteration. We experiment with different learning rates and augmentation of the training set with flipped point cloud sequences which we discuss later in this section. Once the training is complete, we use the scene flow prediction from the forward leg of our model to compute the EPE3D, ACC3D Strict, and ACC3D Relax for each experiment.

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.
Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 4321 in the example.  To do this fine the line (around
line 23)
\begin{verbatim}
\setcounter{page}{4321}
\end{verbatim}
where the number 4321 is your assigned starting page.

Make sure the first page is numbered by commenting out the first page being
empty on line 46
\begin{verbatim}
\end{verbatim}

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\subsection{Implementation Details}

\paragraph*{Setup} We fine tune on KITTI dataset in a self-supervised manner using our model. Although KITTI provides ground truth scene flow annotations for the point clouds, we perform this experiment to test the generalization ability of our model. KITTI is a small-scale dataset and therefore to finetune on KITTI effectively, we initialize the network with the parameters of the model pre-trained on large-scale nuScenes dataset in a self-supervised manner as described above. For this set of experiments, we again train the model for 10K epochs with a batch size of 8. Since KITTI has scenes with two point clouds each, we simply pick the two point clouds as input to the model in each iteration. We experiment with different learning rates and augmentation of the training set with flipped point cloud sequences which we discuss later in this section. Once the training is complete, we use the scene flow prediction from the forward leg of our model to compute the EPE3D, ACC3D Strict, and ACC3D Relax for each experiment.

\section{Introduction}

For an autonomous vehicle, understanding the dynamics of the surrounding environment is critical to ensure safe planning and navigation. It is essential for a self-driving vehicle to be able to perceive the actions of various entities around it, such as other vehicles, pedestrians, and cyclists. In the context of data recorded as 3D point clouds, a motion can be estimated for each 3D point; this is known as scene flow, which refers to the 3D velocity of each 3D point in a scene.
Its 2D analog, optical flow, is the projection of scene flow onto the image plane of a camera. An alternative to scene flow estimation is to use 3D object detection for object-level tracking and to assume that all points within a bounding box have the same rigid motion. However, in such a pipeline, errors in object detection can lead to errors in tracking. In contrast, scene flow methods can avoid such errors by directly estimating the motion of each 3D point.

Recent state-of-the-art methods learn to estimate the scene flow from 3D point clouds~\cite{liu2019flownet3d,gu2019hplflownet,wang2018deep, pointpwcnet}. However, these methods are fully supervised and require annotated datasets for training.  Such annotations are costly to obtain as they require labeling the motion for every point in a scene. To compensate for the lack of real world data, learning-based methods for scene flow have been trained primarily on synthetic datasets and fine tuned on real world data. This requirement of labeled training data limits the effectiveness of such methods in real world settings.

\begin{figure}[t]
 	\centering
 	\includegraphics[trim={5 10 25 0}, clip,width=0.45\textwidth]{figures/overview_car.pdf}
 	\caption{We use two self-supervised losses to learn scene flow on large unlabeled datasets.  The ``nearest neighbor loss" penalizes the distance between the predicted point cloud (\textcolor{green}{green}) and each predicted point's nearest neighbor in the second point cloud (\textcolor{red}{red}).  To avoid degenerate solutions, we also estimate the flow between these predicted points (\textcolor{green}{green}) in the reverse direction back to the original point cloud (\textcolor{blue}{blue}) to form a cycle. The new predicted points from the cycle (\textcolor{magenta}{purple}) should align with the original points (\textcolor{blue}{blue}); the distance between these two set of points forms our second self-supervised loss: ``cycle consistency."}
 	\label{fig:cycle}
  	\vspace{-1em}
\end{figure}

To overcome this limitation, we propose a self-supervised method for scene flow estimation. Using a combination of two self-supervised losses, we are able to mimic the supervision created by human annotation. Specifically, we use a cycle consistency loss, which  ensures that the scene flow produced is consistent in time (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot we ensure that a temporal cycle ends where it started). We also use a nearest neighbor loss; due to the unavailability of scene flow annotations, we consider the nearest point to the predicted translated point, in the temporally next point cloud, as the pseudo-ground truth association.  Intuitively, the nearest neighbor loss pushes one point cloud to flow toward occupied regions of the next point cloud. We show that this combination of losses can be used to train a scene flow network over large-scale, unannotated datasets containing sequential point cloud data. An overview of our method can be found in Figure~\ref{fig:cycle}.

We test our self-supervised training approach using the neural network architecture of a state-of-the-art scene flow method~\cite{liu2019flownet3d}.  The self-supervision allows us to train this network on large-scale, unlabeled autonomous driving datasets. Our method matches the current state-of-the-art performance when no real world annotations are given. Moreover, our method exceeds the performance of state-of-the-art scene flow estimation methods when combined with supervised learning on a smaller labeled dataset.

\section*{Supplementary}
\subsection{Hyperparameters}
A batch size of 8 is used for all datasets. All models are trained using the Adam optimizer with a learning rate of 1e-4, $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=1e-8$.

\subsection{Further Ablation Studies}
In this section, we extend our ablation study to further evaluate the importance of each component used in our method. Beginning from our full method, we remove a single component to see the change on the evaluation metrics.

The results can be seen in Table~\ref{tbl:ablation}.  Because  ``anchoring" is a modification of the cycle loss, when Cycle Consistency Loss is removed, anchoring must be removed too.  The results in Table~\ref{tbl:ablation} show that each component is important for the performance of our method, and removing any of the components results in a drop in performance. This can especially be seen in the accuracy performance metrics (ACC 0.05 and 0.1), which show large drops when any component is removed.

\begin{table}[h]
\centering
\footnotesize
\setlength\tabcolsep{2pt}
\begin{tabular}{c|c|c|c||c|c|c}
\toprule
\multicolumn{7}{l}{Self-Supervised Training (nuScenes + KITTI)} \\ \midrule
NN Loss & Cycle Loss & Anchor  & Flip & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
           & \checkmark & \checkmark & \checkmark & 0.1768 & 15.90 & 35.81 \\
\checkmark &            &            & \checkmark & 0.1102 & 30.80 & 73.27 \\
\checkmark & \checkmark &  & \checkmark & 0.1493 & 22.97 & 49.58 \\
\checkmark & \checkmark &  \checkmark &  & 0.1072 & 40.03 & 72.20 \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & \textbf{0.1053} & \textbf{46.48} & \textbf{79.42} \\
\hline \hline
\multicolumn{7}{l}{Self-supervised (nuScenes) + Supervised Training (KITTI)} \\ \hline
NN Loss & Cycle Loss & Anchor & Flip & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
          & \checkmark & \checkmark & \checkmark & {0.1572} & {18.50} & {52.80} \\
\checkmark &            &            & \checkmark & {0.1090} & {34.88} & {71.32} \\
\checkmark & \checkmark &  & \checkmark & {0.0932} & {28.18} & {66.10} \\
\checkmark & \checkmark &  \checkmark &  & 0.0926 & 40.69 & 74.50 \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & \textbf{0.0912} & \textbf{47.92} & \textbf{79.63}  \\
\bottomrule
\end{tabular}
\caption{Leave-one-out ablation analysis: We study the effect of removing a single component of self-supervised loss and data augmentation. Top: Models use self-supervised training on nuScenes and KITTI; Bottom: Models use self-supervised training on nuScenes followed by supervised training on KITTI.}\label{tbl:ablation}
\vspace{-1em}
\end{table}

\subsection{Analysis of Point Density vs End Point Error}

We analyze the correlation between local point density and endpoint error. For each point in the point cloud at time $t$, we compute the number of points within a 0.1m radius neighborhood of that point. Each point is binned based on the density of its local neighborhood. For each bin, the mean end-point error (EPE) is computed for both the baseline, trained only synthetic data, and our method, trained on nuScenes and KITTI using our self-supervised losses. Figure \ref{fig:query_ball} shows no correlation between EPE and neighborhood density for either the baseline or for our method.

\begin{figure}
  \includegraphics[width=\linewidth]{figures/plots_query_ball_new.png}
  \caption{Analysis of average EPE (m) with respect to the local point density (points/$m^3$). Flow estimates are binned by local point density and a confidence interval of 95\% is shown for all results.}
  \label{fig:query_ball}
\end{figure}

\subsection{Error Distribution of End Point Error}

We analyze the error distributions of our method and a baseline method. Our method uses self-supervised training on nuScenes followed by supervised training on KITTI. The baseline is trained only on KITTI using supervised learning. By computing the error at every point of every scan in the KITTI test set, we can view the full distribution of end point errors (EPE), shown in Figure~\ref{fig:error_distribution}. To better show the effects of outliers, we use log binning for the x-axis. Not only is the center of our error distribution lower, with an average EPE of 0.091, but it also shows fewer large outliers than the baseline method.

\begin{figure}
  \includegraphics[width=\linewidth]{figures/ours_sup_vs_base_sup_final.png}
  \caption{End point error distribution of our method (\textcolor{blue}{blue}) and the baseline (\textcolor{orange}{orange}).}
  \label{fig:error_distribution}
\end{figure}
\section{Conclusion}

In this work, we propose a self-supervised method for training scene flow algorithms using a combination of cycle consistency in time and nearest neighbor losses. Our purely self-supervised method is able to achieve a performance comparable to that of the supervised methods on the widely used KITTI self-driving dataset. We further show that when supervised training is augmented with self-supervision on a large-scale, unannotated dataset, the results exceed the current state-of-the-art performance. Our self-supervision method opens the door to fine-tuning on arbitrary datasets that lack scene flow annotations.

\subsection*{Acknowledgements}
This work was supported by the CMU Argo AI Center for Autonomous Vehicle Research and a NASA Space Technology Research Fellowship.
\section{Related Work}
\paragraph*{Scene Flow} Vedula \etal~\cite{vedula1999three} first introduced the task of scene flow estimation. They propose a linear algorithm to compute it from optical flow. Other works involve joint optimization of camera extrinsics and depth estimates for stereo scene flow~\cite{valgaerts2010joint}, use of particle filters~\cite{hadfield2011kinecting}, local rigid motion priors~\cite{vogel20113d,vogel20153d,vogel2013piecewise, menze2015object}, and smoothness-based regularization~\cite{basha2013multi}.

\vspace{-1em}
\paragraph*{Deep Scene Flow}
State-of-the-art scene flow methods today use deep learning to improve performance.
FlowNet3D~\cite{liu2019flownet3d} builds on PointNet++~\cite{qi2017pointnet++, qi2017pointnet} to estimate scene flow directly from a pair of point clouds.
Gu \etal ~\cite{gu2019hplflownet} produced similar results using a permutohedral lattice to encode the point cloud data in a sparse, structured manner.  The above approaches compute scene flow directly from 3D point clouds~\cite{liu2019flownet3d,gu2019hplflownet}. Methods involving voxelizations with object-centeric rigid body assumptions~\cite{Behl_2019_CVPR}, range images~\cite{vaquero2018deep}, and non-grid structured data~\cite{wang2018deep} have also been used for scene flow estimation. All of the above methods were trained either with synthetic data~\cite{mayer2016large} or with a small amount of annotated real-world data~\cite{kitti} (or both).  Our self-supervised losses enable training on large unlabeled datasets, leading to large improvements in performance.

\vspace{-1em}
\paragraph*{Self-Supervised Learning} Wang \etal ~\cite{wang2019learning} used self-supervised learning for 2D tracking on video. They propose a tracker which takes a patch of an image at time $t$ and the entire image at time $t-1$ to track the image patch in the previous frame. They define a self-supervised loss by tracking the patch forward and backward in time to form a cycle while penalizing the errors through cycle consistency and feature similarity. We take inspiration from this work for our self-supervised flow estimation on point clouds. Other works including self-supervisory signals are image frame ordering~\cite{misra2016shuffle}, feature similarity over time~\cite{wang2015unsupervised} for images, and clustering and reconstruction~\cite{hassani2019ICCV} from point clouds. While these can potentially be used for representation learning from 3D data, they cannot be directly used for scene flow estimation. Concurrent to our work, Wu \etal~\cite{pointpwcnet} showed that Chamfer distance, smoothness constraints, and Laplacian regularization can be used to train scene flow in a self-supervised manner.

\section{Experiments}

We run several experiments to validate our self-supervised method for scene flow estimation for various levels of supervision and different amounts of data. First, we show that our method, with self-supervised training on large unlabeled datasets, can perform as well as supervised training on the existing labeled data.
Next, we investigate how our results can be improved by combining self-supervised learning with a small amount of supervised learning, exceeding the performance of purely supervised learning. Finally, we explore the utility of each element of our method through an ablation study.

\subsection{Implementation Details}
For all data configurations (our method and the baseline), we initialize our network with the parameters of the Flownet3D model~\cite{liu2019flownet3d} pre-trained on the FlyingThing3D dataset~\cite{mayer2016large}. We compare our self-supervised training procedure to a baseline which uses supervised fine-tuning on the KITTI dataset~\cite{kitti}.  The baseline used in the comparison is the same as in Liu \etal~\cite{liu2019flownet3d}, except that we increase the number of training iterations from 150 epochs (as described in the original paper) to 10k epochs in order to keep the number of training iterations consistent with that used in our self-supervised method. We see that this change leads to a small improvement in the baseline performance, which we include in the results table.

\begin{figure*}[t]
 	\centering
 	\includegraphics[trim={0 0 0 50}, clip, width=\textwidth]{figures/self-supervised-unsupervised-f3d.pdf}
 	\caption{Scene flow estimation between point clouds at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green}) from the KITTI dataset trained without any labeled LIDAR data. Predictions from our self-supervised method, trained on nuScenes and fine-tuned on KITTI using self-supervised learning is shown in \textcolor{blue}{blue}; the baseline with only synthetic training is shown in \textcolor{magenta}{purple}. In the absence of real-world supervised training, our method clearly outperforms the baseline method, which overestimate the flow in many regions. (Best viewed in color)}
 	\label{fig:qual_no_ground_truth}
\end{figure*}

\begin{figure*}[t]
 	\centering

 	\begin{subfigure}{0.48\textwidth}
 	\includegraphics[trim={0 0 900 50}, clip, width=\textwidth]{figures/nuscenes_comparison.pdf}
 	\caption{Ours (Self-Supervised Fine Tuning)}
 	\end{subfigure}

  	\begin{subfigure}{0.48\textwidth}
 	\includegraphics[trim={920 0 0 50}, clip, width=\textwidth]{figures/nuscenes_comparison.pdf}
 	\caption{Baseline (No Fine Tuning)}
 	\end{subfigure}
 	\caption{Comparison of our self-supervised method to a baseline trained only on synthetic data, shown on the nuScenes dataset. Scene flow is computed  between point clouds at time $t$~(\textcolor{red}{red}) and $t+1$~(\textcolor{green}{green}); the point cloud that is transformed using the estimated flow is in shown in \textcolor{blue}{blue}. In our method, the predicted point cloud has a much better overlap with the point cloud of the next timestamp (\textcolor{green}{green})  compared to the baseline. Since nuScenes dataset does not provide any scene flow annotation, the supervised approaches cannot be fined tuned to this environment.}
 	\label{fig:qual_nuscenes}
 	\vspace{-1em}
\end{figure*}

\begin{figure*}[ht!]
 	\centering
 	\includegraphics[trim={0 0 0 40}, clip, width=\textwidth]{figures/supervised_new.pdf}
 	\caption{Scene flow estimation on LIDAR data from the KITTI dataset between point clouds at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green}). Our method, which is trained on nuScenes using self-supervised learning and then fine-tuned on KITTI using supervised learning, is shown in \textcolor{blue}{blue}. The baseline method is fine-tuned only on KITTI using supervised training and is shown in \textcolor{magenta}{purple}. While in aggregate, both methods well estimate the scene flow, adding self-supervised training on nuScenes (\textcolor{blue}{blue}) enables our predictions to more closely match the next frame point cloud (\textcolor{green}{green}). In several  scenes, the purely supervised method (\textcolor{magenta}{purple}) underestimates the flow, staying too close to the initial point cloud (\textcolor{red}{red}). (Best viewed in color)}
 	\label{fig:qual_supervised}

\end{figure*}

\subsection{Datasets}

\paragraph*{KITTI Vision Benchmark Suite}  KITTI~\cite{kitti} is a real-world self-driving dataset. There are 150 scenes of LIDAR data in KITTI collected using seven scans of a Velodyne 64 LIDAR, augmented using 3D models, and annotated with ground truth scene flow~\cite{menze2018object}. For our experiments under both self-supervised and supervised settings, we consider 100 out of 150 scenes for training and the remaining 50 scenes for testing. Ground points are removed from every scene using the pre-processing that was performed in previous work~ \cite{liu2019flownet3d}. Every scene consists of a pair of point clouds recorded at two different times as well as the ground truth scene flow for every point of the first point cloud.

\vspace{-1em}
\paragraph*{nuScenes} The nuScenes~\cite{nuscenes2019} dataset is a large-scale public dataset for autonomous driving. It consists of 850 publicly available driving scenes in total from Boston and Singapore. The LIDAR data was collected using a Velodyne 32 LIDAR rotating at 20 Hz. This is in contrast to the 64-beam Velodyne rotating at 10 Hz used for the KITTI dataset.  This difference in sensors leads to a difference in data sparsity that creates a distribution shift between KITTI and nuScenes.  This distribution shift necessitates additional training on KITTI beyond our self-supervised training on nuScenes.  Nonetheless, our results show a substantial benefit from the self-supervised training on nuScenes.

Since the nuScenes dataset~\cite{nuscenes2019} does not contain scene flow annotations, we must use self-supervised methods when working with this dataset. In our experiments, out of the 850 scenes available, we use 700 as the train set and the rest 150 as the validation set. Similar to KITTI, we remove the ground points from each point cloud using a manually tuned height threshold.

\begin{table*}[h]
\centering
\begin{tabular} {l|c c c}
\toprule
 Training Method & EPE (m) $\downarrow$ & ACC (0.05) $\uparrow$ & ACC (0.1) $\uparrow$ \\
\hline
\hline
No Fine Tuning & 0.122 & 25.37\% & 57.85\% \\
KITTI (Supervised) &
0.100 & 31.42\% & 66.12\% \\
\hline
Ablation: KITTI (Self-Supervised)  & 0.126 & 32.00\% & 73.64\% \\
Ours: nuScenes (Self-Supervised) + KITTI (Self-Supervised)  & 0.105 & 46.48\% & 79.42\% \\
Ours: nuScenes (Self-Supervised) + KITTI (Supervised) & \textbf{0.091} & \textbf{47.92\%} & \textbf{79.63\%} \\
\bottomrule
\end{tabular}
\caption{Comparison of levels of supervision on KITTI dataset. The nearest neighbor + anchored cycle loss is used for nuScenes (self-supervised) and KITTI (self-supervised). All methods are pretrained on FlyingThings3D\cite{mayer2016large} and ground points are removed for KITTI and nuScenes datasets.}\label{tbl:supervision}
\vspace{-1em}
\end{table*}

\subsection{Results}
We use three standard metrics to quantitatively evaluate the predicted scene flow when the ground truth annotations of scene flow are available. Our primary evaluation metric is End Point Error (EPE) which describes the mean Euclidean distance between the predicted and ground truth transformed points, described by Equation~\ref{equ:gt}. We also compute accuracy at two threshold levels, Acc(0.05) as the percentage of scene flow prediction with an EPE~$<~0.05$m or relative error~$<5\%$ and Acc(0.1) as percentage of points having an EPE~$<0.1$m or relative error~$<~10\%$, as was done for evaluation in previous work~\cite{liu2019flownet3d}.

\subsubsection{Quantitative Results}

\paragraph*{Self-supervised training:}
Unlike previous work, we are not restricted to annotated point cloud datasets; our method can be trained on any sequential point cloud dataset. There are many point cloud datasets containing real LIDAR captures of urban scenes, but most of them do not contain scene flow annotations. Due to lack of annotations, these datasets can not be utilized for supervised scene flow learning. In contrast, our self-supervised loss allows us to easily integrate them into our training set. The combination of these datasets (KITTI + NuScenes) contains 5x more real data than using KITTI alone.

The results are shown in Table~\ref{tbl:supervision}.
To show the value of self-supervised training, we evaluate the performance of our method without using any  ground-truth annotations.  We first pre-train on the synthetic FlyingThings3D dataset; we then perform self-supervised fine-tuning on the large nuScenes dataset followed by further self-supervised fine-tuning on the smaller KITTI dataset (4th row: ``Ours: nuScenes (Self-Supervised) + KITTI (Self-Supervised)").  As can be seen,
using no real-world annotations, we are able to achieve an EPE of 0.105~m.  This outperforms the baseline of only training on synthetic data (``No Fine Tuning").  Even more impressively, our approach performs similarly to the baseline which
pre-trains on synthetic data and then
does supervised fine-tuning  on the KITTI dataset (``KITTI (Supervised)"); our method has a similar EPE and outperforms this baseline in terms of accuracy, despite not having access to any annotated training data. This result shows that our method for self-supervised training, with a large enough unlabeled dataset, can match the performance of supervised training.

\vspace{-1em}
\paragraph*{Self-supervised + Supervised:} Finally, we show the value of combining our self-supervised learning method with a small amount of supervised learning. For this analysis, we perform self-supervised training on NuScenes as above, followed by supervised training on the much smaller KITTI dataset.  The results are shown in the last row of Table~\ref{tbl:supervision}.

As can be seen, this approach of self-supervised training followed by supervised fine-tuning outperforms all other methods on this benchmark, obtaining an EPE of 0.091, outperforming the previous state-of-the-art result which used only supervised training.  This shows the benefit of self-supervised training on large unlabeled datasets to improve scene flow accuracy, even when scene flow annotations are available.

While Table~\ref{tbl:supervision} only shows results using the FlowNet3D~\cite{liu2019flownet3d} architecture, we note that our method also outperforms the results of HPLFlownet~\cite{gu2019hplflownet}~(EPE of 0.1169) and all models they compare against as well.

Figure \ref{fig:epe_vs_mag} provides an analysis on the correlation between average endpoint error and the magnitude of the ground truth flow.  As can be seen, our method consistently outperforms the baseline at almost all flow magnitudes.

\begin{figure}[t]
 	\centering
 	\includegraphics[width=0.47\textwidth]{figures/self-super-vs-unsuper-new.png}
 	\caption{Analysis of average EPE (m) with respect to ground truth flow magnitudes (m). Flow estimates are binned by ground truth flow magnitude and a confidence interval of 95\% is shown for all results.}
 	\label{fig:epe_vs_mag}
 	\vspace{-1em}
\end{figure}

\subsubsection{Qualitative Analysis}
\paragraph*{Self-supervised training - KITTI results:}
Next we perform a qualitative analysis to visualize the performance of our method.
We compare our method (synthetic pre-training + self-supervised training on nuScenes + self-supervised training on KITTI) compared to the baseline of synthetic training only.  Results on KITTI are shown in Figure~\ref{fig:qual_no_ground_truth}. The figure shows the point clouds captured at time $t$ and $t+1$  in red and green, respectively. The predictions from our method are shown in blue and the baseline predictions are shown in purple. As shown, our scene flow predictions (blue)  have a large overlap with the point cloud at time $t+1$ (green). On the other hand, the baseline predictions (purple) do not overlap with the point cloud at time $t+1$.  The baseline, trained only on synthetic data, fails to generalize to the real-world KITTI dataset. On the contrary, our self-supervised approach can be fine tuned on any real world environment and shows a significant improvement over the baseline.

\vspace{-1em}
\paragraph*{Self-supervised training - nuScenes results:}

Next, we visualize the performance of our method on the nuScenes dataset.  Note that, because nuScenes does not have scene flow annotations, only qualitative results can be shown on this dataset. For this analysis, our method is pre-trained on synthetic data (FlyingThings3D) as before and then fine-tuned on nuScenes in a self-supervised manner. No scene flow annotations for nuScenes are available, so we compare to a baseline which is trained only on synthetic data.

The results on nuScenes are shown in Figure~\ref{fig:qual_nuscenes}. These results again showcase the advantages of self-supervision on real world data over purely synthetic supervised training. As before, the figure shows the point clouds captured at time $t$ and $t+1$ in red and green respectively. The predictions are shown in blue, with our method on the left (Fig.~\ref{fig:qual_nuscenes}a) and the baseline on the right (Fig.~\ref{fig:qual_nuscenes}b).  As shown, our scene flow predictions (left figure, blue) have a large overlap with the point cloud at time $t+1$ (green). On the other hand, the baseline predictions (right figure, blue) do not overlap with the point cloud at time $t+1$.

The low performance of the baseline can again be attributed to its training on synthetic data and its inability to generalize to real-world data.  For the nuScenes data, no scene flow annotations exist, so only self-supervised learning is feasible to improve performance.

\begin{table}[]
\centering
\footnotesize
\setlength\tabcolsep{2pt}
\begin{tabular}{c|c|c|c||c|c|c}
\toprule
\multicolumn{7}{l}{Self-Supervised Training (nuScenes + KITTI)} \\ \midrule
NN Loss & Cycle Loss & Anchor Pts & Flip & EPE (m) & ACC (0.05) & ACC (0.1) \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & 0.105 & 46.48\% & 79.42\% \\
\checkmark & \checkmark & \checkmark &  & 0.107 & 40.03\% & 72.20\% \\
\checkmark & \checkmark &  &  & 0.146 & 30.21\% & 48.57\% \\
\checkmark &  &  &  & 0.108 & 42.00\% & 78.51\% \\
 &  &  &  & 0.122 & 25.37\% & 57.85\% \\
\hline \hline
\multicolumn{7}{l}{Self-Supervised (nuScenes) + Supervised Training (KITTI)} \\ \hline
NN Loss & Cycle Loss & Anchor Pts & Flip & EPE (m) & ACC (0.05) & ACC (0.1) \\
\hline
\checkmark & \checkmark & \checkmark & \checkmark & {0.091} & {47.92\%} & {79.63\%} \\
\checkmark & \checkmark & \checkmark &  & {0.093} & {40.69\%} & {74.50\%} \\
\checkmark & \checkmark &  &  & 0.092 & 30.76\% & 72.94\% \\
\checkmark &  &  &  & 0.114 & 31.24\% & 64.58\% \\
 &  &  &  & 0.100 & 31.42\% & 66.12\%\\ \bottomrule
\end{tabular}
\caption{Ablation analysis: We study the effect of the different self-supervised losses and data augmentation. Top: Models use self-supervised training on nuScenes and KITTI; Bottom: Models use self-supervised training on nuScenes followed by supervised training on KITTI.}\label{tbl:ablation}
\vspace{-1em}
\end{table}

\begin{table}
\centering
\footnotesize
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c | c | c | c}
\toprule
$\lambda$ & EPE (m)$\downarrow$ & ACC (0.05)$\uparrow$ & ACC (0.1)$\uparrow$ \\
\hline
0   & 0.120 & 24.09\% & 73.20\% \\
0.25 & 0.122 &  26.41\% & 65.57\% \\
0.5 & \textbf{0.105} & \textbf{46.48\%} & \textbf{79.42\%} \\
0.75 &  0.125 & 23.59\% & 62.96\% \\
1   & 0.149 & 22.97\% & 49.58\% \\
\bottomrule
\end{tabular}
\caption{Effect of varying the $\lambda$ parameter for ``anchoring" the Cycle Consistency Loss. Results are shown for self-supervised training on nuScenes + KITTI.}\label{tbl:cycle_lambda}
\vspace{-1em}
\end{table}

\vspace{-1em}
\paragraph*{Self-supervised + supervised training - KITTI results:}
Finally, we show the value of combining our self-supervised learning method with a small amount of supervised learning, compared to only performing supervised learning.  For our method, we perform synthetic pre-training, followed by self-supervised fine-tuning on nuScenes, followed by supervised fine-tuning on the much smaller KITTI dataset. We compare to a baseline which only performs synthetic pre-training followed by supervised fine-tuning KITTI.

Qualitative results can be seen in Figure~\ref{fig:qual_supervised}. The figure shows the point clouds captured at time $t$ and $t+1$  in red and green, respectively. The predictions from our method are shown in blue and the baseline predictions are shown in purple. As shown, our scene flow predictions (blue)  have a large overlap with the point cloud at time $t+1$ (green), whereas the baseline predictions (purple) do not.

The baseline predicts a small motion, keeping the transformed cloud (purple) too close to the initial position (red). As discussed above, this bias towards small motion is likely due to the training of the baseline over a synthetic dataset, which affects the generalization of the baseline to real-world datasets where objects exhibit different types of motion than seen in simulation. By training over a significantly larger unlabeled dataset, our method is able to avoid overfitting and generalizes better to the scenes and flow patterns which were not present in the synthetic dataset.

\begin{figure}[t]
 	\centering
 	\includegraphics[trim={0 0 0 100}, clip, width=\columnwidth]{figures/cycle_vs_forward_comparison.pdf}
 	\caption{Ablation study comparing our self-supervised method with both nearest neighbor loss and anchored cycle consistency loss (\textcolor{blue}{blue}) compared to training only using the nearest neighbor loss (\textcolor{magenta}{purple}). Scene flow is computed between point clouds from the KITTI dataset at time $t$ (\textcolor{red}{red}) and $t+1$ (\textcolor{green}{green})}
 	\label{fig:nuscenes_ablation}
  	\vspace{-1em}
\end{figure}

\subsubsection{Ablation Studies}

We test the importance of each component of our method by running a series of ablation studies. Table~\ref{tbl:ablation} shows the effect of iteratively removing portions of our method in both the purely self-supervised (top) and self-supervised + supervised (bottom) training.
The benefits of anchored cycle consistency loss (compared to only using nearest neighbor loss) can be seen in  Table~\ref{tbl:ablation} (bottom) as well as in Figure~\ref{fig:nuscenes_ablation}.

The benefits of anchoring is apparent by the large drop in performance for self-supervised training (top) when anchoring is removed.
 Introducing the anchor point cloud as part of the backward flow greatly improves performance when only self-supervised training is used (Figure~\ref{fig:nuscenes_ablation}, top).
  This suggests that having anchored point clouds stabilizes the training of the cycle consistency loss. Additionally, we evaluate the sensitivity of our system to the selection of the anchoring parameter, $\lambda$. Table~\ref{tbl:cycle_lambda} shows that we obtain the best results with $\lambda=0.5$, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot using the average of the predicted and nearest points. Overall, these analyses show the benefits of each component of our method. Further ablation analysis can be found in the supplement.

\section{Method}

\subsection{Problem Definition}
For the task of scene flow estimation, we have a temporal sequence of point clouds: point cloud $\mathcal{X}$ as the point cloud captured at time $t$ and point cloud $\mathcal{Y}$ captured at time $t+1$. There is no structure enforced on these point clouds and they can be recorded directly from a LIDAR sensor or estimated through a stereo algorithm.
Each point $p_i = \{x_i,f_i\}$ in point cloud $\mathcal{X}$ contains the Cartesian position of the point, $x_i \in \mathbb{R}^3$, as well as any additional information which the sensor produces, such as color, intensity, normals, etc, represented by $f_i \in \mathbb{R}^c$.

The scene flow, $\mathcal{D} = \{d_i\}^N, d_i \in \mathbb{R}^3$ between these two point clouds describes the movement of each point $x_i$ in point cloud $\mathcal{X}$ to its corresponding position $x_i'$ in the scene described by point cloud $\mathcal{Y}$, such that $x_i' = x_i + d_i$,  and $N$ is the size of point cloud $\mathcal{X}$. Scene flow is defined such that $x_i$ and $x'_i$ represent the same 3D point of an object moved in time. Unlike optical flow estimation, the exact 3D position of $x_i'$ may not necessarily coincide with a point in the point cloud $\mathcal{Y}$, due to the sparsity of the point cloud. Additionally, the sizes of $\mathcal{X}$ and $\mathcal{Y}$ may be different.

\vspace{-1em}
\paragraph*{Supervised Loss}
The true error associated with our task is the difference between the estimated flow $g(\mathcal{X}, \mathcal{Y}) = \mathcal{\hat{D}} = \{\hat{d}_i\}^N$ and the ground truth flow $\mathcal{D}^* = \{d^*_i\}^N$,
\begin{align}
\mathcal{L}_{gt} = \frac{1}{N} \sum_i^N \|d_i^*-\hat{d}_i\|^2.
\label{equ:gt}
\end{align}
The loss in Equation~\ref{equ:gt} is useful because it is mathematically equivalent to the end point error, which we use as our evaluation metric. However, computing this loss requires annotated ground truth flow $d^*_i$. This type of annotation is easy to calculate in synthetic data~\cite{mayer2016large}, but requires expensive human labeling for real world datasets. As such, only a small amount of annotated scene flow datasets are available ~\cite{menze2015joint, menze2018object}. While training on  purely synthetic data is possible, large improvements can often be obtained by training on real data from the domain of the target application.  For example, Lui \etal~\cite{liu2019flownet3d} showed an 18\% relative improvement after fine-tuning on a small amount of annotated real world data.  This result motivates our work to use self-supervised training to train on large unlabeled datasets.

\begin{figure}[t]
 	\centering
 	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={70 250 70 250}, clip, width=\linewidth]{figures/car_loss_forward_30.pdf}

 	\caption{Nearest Neighbor Loss}
 	\end{subfigure}

  	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={70 250 70 250}, clip, width=\linewidth]{figures/car_loss_cycle_30.pdf}
 	\caption{Cycle Consistency Loss}
 	\end{subfigure}
 	\caption{Example of our self-supervised losses between consecutive point clouds $\mathcal{X}$ (\textcolor{blue}{blue}) and $\mathcal{Y}$ (\textcolor{red}{red}). We consider the point $x$ whose ground truth projected point $x'$ is not known during training time. $(a)$ Nearest Neighbor Loss is computed between the projected point $\hat{x}'$, predicted by the forward flow, and the closest point in $\mathcal{Y}$.  $(b)$ The Cycle Consistency Loss tracks this transformed point back onto its original frame, as point $\hat{x}''$, using the reverse flow, and computes the distance to its original position ${x}$.}
 	\label{fig:losses}
  	\vspace{-1em}
\end{figure}

\vspace{-1em}
\paragraph*{Nearest Neighbor (NN) Loss}

For large unlabeled datasets, since we do not have information about $d_i^*$, we cannot compute the loss in Equation~\ref{equ:gt}.
In lieu of annotated data, we take inspiration from methods such as Iterative Closest Point~\cite{besl1992method} and use the nearest neighbor of our transformed point $\hat{x}_i' = x_i + \hat{d}_i$ as an approximation for the true correspondence. For each transformed point $\hat{x}_i' \in \hat{\mathcal{X}}'$, we find its nearest neighbor $y_j \in \mathcal{Y}$ and compute the Euclidean distance to that point, illustrated as $e_{NN}$ in Figure~\ref{fig:losses}a:
\begin{equation}
    \mathcal{L}_{NN} = \frac{1}{N}\sum_i^N \min_{y_j \in \mathcal{Y}}\|\hat{x}_i' - y_j\|^2.
\label{equ:nn}
\end{equation}

Assuming that the initial flow estimate is sufficiently close to the correct flow estimate, this loss will bring the transformed point cloud and the target point cloud closer. This loss can, however, have a few drawbacks if imposed alone.  First, the true position of the point $x_i$ transformed by the ground truth flow, $x_i' = x_i + d_i^*$, may not be the same as the position of the nearest neighbor to $\hat{x}'$ (transformed by the \emph{estimated} flow) due to potentially large errors in the estimated flow, as illustrated in Figure~\ref{fig:losses}a.  Further, the position of $x_i'$ may not correspond with any point in $\mathcal{Y}$ if the point cloud $\mathcal{Y}$ is sufficiently sparse, as is common for point clouds collected by sparse 3D LIDAR for autonomous driving.  Finally, this loss does not penalize degenerate solutions where all of the points in $\mathcal{X}$ map to the same point in $\mathcal{Y}$; such a degenerate solution would obtain 0 loss under Equation~\ref{equ:nn}. To address these issues, we use an additional self-supervised loss: cycle consistency loss.

\vspace{-1em}
\paragraph*{Cycle Consistency Loss}
To avoid the above issues, we incorporate an additional self-supervised loss: cycle consistency loss, illustrated in Figure~\ref{fig:losses}b. We first estimate the ``forward" flow as $\hat{\mathcal{D}} = g(\mathcal{X}, \mathcal{Y})$.  Applying the estimated flow $\hat{d}_i \in \hat{\mathcal{D}}$ to each point $x_i \in \mathcal{X}$ gives an estimate of the location of the point $x_i$ in the next frame: $\hat{x}_i' = x_i + \hat{d}_i$.  We then compute the scene flow in the reverse direction: for each transformed point $\hat{x}_i'$ we estimate the flow to transform the point back to the original frame, $\hat{\mathcal{D'}} = g(\mathcal{\hat{X}'}, \mathcal{X})$.  Transforming each point $\hat{x}_i'$ by this ``reverse" flow $\hat{d}_i'$ gives a new estimated point $\hat{x}_i''$.  If both the forward and reverse flow are accurate, this point $\hat{x}_i''$ should be the same as the original point $x_i$.  The error between these points, $e_{cycle}$, is the ``cycle consistency loss,"  given by
\begin{equation}
    \mathcal{L}_{cycle} = \sum_i^N \|\hat{x}_i'' - x_i\|^2.
\label{equ:cycle}
\end{equation}
A similar loss is used as a regularization in~\cite{liu2019flownet3d}.

However, we found that implementing the cycle loss in this way can produce unstable results if only self-supervised learning is used without any ground-truth annotations.  These instabilities appear to be caused by errors in the estimated flow which lead to structural distortions in the transformed point cloud $\hat{\mathcal{X}}'$, which is used as the input for computing the reverse flow $g(\mathcal{\hat{X}'}, \mathcal{X})$. This requires the network to simultaneously learn to correct any distortions in $\mathcal{\hat{X}'}$, while also learning to estimate the true reverse flow.
To solve this problem, we use the nearest neighbor $y_j$ of the transformed point $\hat{x}'_i$ as an anchoring point in the reverse pass.  Using the nearest neighbor $y_j$ stabilizes the structure of the transformed cloud while still maintaining the correspondence around the cycle. The effects of this stabilization are illustrated in Figure~\ref{fig:anchor}. As we are using the anchoring point as part of the reverse pass of the cycle, we refer to this loss as ``anchored cycle consistency loss''.

Specifically, we compute the anchored reverse flow as follows. First, we compute the forward flow as before, $\hat{\mathcal{D}} = g(\mathcal{X}, \mathcal{Y})$, which we use to compute the transformed point cloud $\hat{x}_i' = x_i + \hat{d}_i$.  We then compute anchor points $\bar{\mathcal{X}'} = \{\bar{x}_i'\}^N$ as a convex combination of the transformed point and its nearest neighbor $\bar{x}_i' = \lambda \hat{x}_i' + (1-\lambda) y_j$.  In our experiments, we find that $\lambda=0.5$ produces the most accurate results. Finally, we compute the reverse flow using these anchored points: $\bar{\mathcal{D}}' = g(\bar{\mathcal{X}}', \mathcal{X})$.  The cycle loss of Equation~\ref{equ:cycle} is then applied to this anchored reverse flow.  By using anchoring, some of the structural distortion of the predicted point cloud $\mathcal{\hat{X}'}$ will be removed in the anchored point cloud $\bar{\mathcal{X}'}$, leading to a more stable training input for the reverse flow.

Note that the cycle consistency loss also has a degenerate solution: the ``zero flow,", \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot $\hat{\mathcal{D}} = 0$, will produce 0 loss according to Equation~\ref{equ:cycle}.  However, the zero flow will produce produce a non-zero loss when anchored cycle consistency is used; thus anchoring helps to remove this degenerate solution.  Further, the nearest neighbor loss will also be non-zero for the degenerate solution of zero flow.  Thus, the local minima of each of the nearest neighbor and cycle consistency losses conflict, allowing their sum, $\mathcal{L} = \mathcal{L}_{NN} + \mathcal{L}_{cycle}$, to act as a stable surrogate for the true error.

\begin{figure}[t]
 	\centering
 	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={20 70 20 70}, clip, width=\linewidth]{figures/car_noise.png}
 	\caption{}
 	\end{subfigure}
 	\hspace*{\fill}
  	\begin{subfigure}{0.23\textwidth}
 	\includegraphics[trim={20 70 20 70}, clip, width=\linewidth]{figures/car_anchor.png}
 	\caption{}
 	\end{subfigure}
 	\caption{Compounding errors cause problems in estimating reverse flow using the transformed point cloud. $(a)$ Large flow prediction errors degrade the structure of the transformed cloud $\hat{\mathcal{X}}'$ (shown in \textcolor{green}{green}). Thus, computing the reverse flow between $\hat{\mathcal{X}}'$ (\textcolor{green}{green}) and $\mathcal{X}$ (\textcolor{blue}{blue}) is an ill-posed task.
 	$(b)$ Using the nearest neighbor points (\textcolor{red}{red}) as anchors, we are able to stabilize the transformed cloud $\bar{\mathcal{X}}'$ (\textcolor{cyan}{cyan}), thus retaining important structural information.}
 	\label{fig:anchor}
  	\vspace{-1em}
\end{figure}

\vspace{-1em}
\paragraph*{Temporal Flip Augmentation}

Having a dataset of point cloud sequences in only one direction may generate a motion bias which may lead to the network predicting the flow equal to the average forward speed of the training set. To reduce this bias, we augment the training set by flipping the point clouds, i.e, reversing the flow. With this augmentation, the network sees an equal number of point cloud sequences having a forward motion and a backward motion.

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.
Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 4321 in the example.  To do this fine the line (around
line 23)
\begin{verbatim}
\setcounter{page}{4321}
\end{verbatim}
where the number 4321 is your assigned starting page.

Make sure the first page is numbered by commenting out the first page being
empty on line 46
\begin{verbatim}
\end{verbatim}

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv