text:
\section{Disparity Estimation}
\label{sec: approach}

\textit{For brevity, in the following we mainly discuss disparity estimation. The same technique can easily be applied to depth estimation, which is usually adapted from their disparity estimation counterparts.}

As reviewed in~\autoref{sec: related}, many existing stereo networks output a distribution of disparities at each pixel.
This distribution is a categorical distribution over discrete disparity values: discrete because they are estimated as the difference in $X$-coordinates of corresponding pixels, and as such are integers.
Stereo techniques then compute the mean of the distribution to obtain a continuous estimate that is not limited to integral values.

\begin{wrapfigure}{r}{0.7\textwidth}
\vskip -17pt
\includegraphics[width=\linewidth]{figures/distribution.pdf}
\vskip -5pt
\caption{\small \textbf{The predicted disparity posterior for a pixel on object boundaries.} The uni-modal assumption can break down, leading to a  \emph{mean} estimate that is in a low probability region. Learning offsets allow us to predict the continuous \emph{mode}. (Offsets are in $[0, 1]$ here.)
} \label{fig:depth_pred}
\vskip -5pt
\end{wrapfigure}

We point out two disadvantages of taking the mean estimate. First, the mean value can deviate from the mode and may wrongly predict values of low probability when the predicted distribution is multi-modal (see~\autoref{fig:depth_pred}). Such multi-modal distributions appear frequently at pixels around the object boundaries. While they collectively occupy only a tiny portion of image pixels, recent studies have shown their particular importance in the downstream tasks like 3D object detection~\cite{li2019stereo,qian2020end,li2018stereo}.
For instance, let us consider a street scene where a car $30$m away (a disparity of, say, 10 pixels) is driving on the road towards the camera, with the sky as the background. The pixels on the car boundary can either take a disparity of around 10 pixels (for the car) or a disparity of 0 pixels (for the sky). Simply taking the mean likely produces arbitrary disparity estimates between these values, producing depth estimates that are neither on the car nor on the background. The downstream 3D object detector can, therefore, wrongly predict the car orientation and size, potentially leading to accidents. Second,
the physical meaning of the mean value is by no means aligned with the true disparity: uncertainty in correspondence might yield a $40\%$ chance of a disparity of 10 pixels and a $60\%$ chance for a disparity of 20 pixels,  but this does not mean that the disparity should be 16 pixels.

Instead, a more straightforward way to simultaneously model the uncertainty and output continuous disparity estimates is to extend the support of the output distribution beyond integers.

\subsection{Continuous disparity network (CDN)}
\label{ssec:CDN}
To this end, we propose a new neural network architecture and output representation for disparity estimation.
The output of our network will still be a set of discrete values with corresponding probabilities, but the discrete values \emph{will not be restricted to integers}.
The key idea is to start with integral disparity values, and predict \emph{offsets} in addition to probabilities.

Denote by $\mathcal{D}$ the set of integral disparity values.
As above, disparity estimation techniques produce a cost $S_\text{disp}(u,v,d)$ for every $d \in \mathcal{D}$.
A softmax converts this cost into a probability distribution:
\begin{align}
p(d|u,v) = \left\{\begin{array}{lr} \softmax(-S_\text{disp}(u,v,d)) & \textrm{if } d \in \mathcal{D}, \\ 0 & \textrm{otherwise}. \end{array} \right. \label{eq_cat}
\end{align}

We propose to add a sub-network $b(u,v,d)$ that predicts an offset disparity value for each integral disparity value $d\in\mathcal{D}$ at each pixel $(u,v)$.
We use this to displace the probability mass at $d \in \mathcal{D}$ to $d' = d + b(u,v,d)$.
This results in the following probability distribution:
\iffalse
\begin{align}
\tilde{p}(d'|u,v) = \left \{ \begin{array}{lr} \softmax(-S_\text{disp}(u,v,d)) & \textrm{when } d' = d + b(u,v,d), d \in \mathcal{D}, \\ 0 & \textrm{otherwise}. \end{array} \right. \label{eq_cont}
\end{align}
\fi
\begin{align}
\tilde{p}(d'|u, v) = \sum_{d\in\mathcal{D}} p(d|u,v) \delta(d' - (d + b(u,v,d))), \label{e_mix_Dirac}
\end{align}
which is a mixture of Dirac delta functions over arbitrary disparity values $d'$.
In other words, $\tilde{p}$ has $|\mathcal{D}|$ supports, each located at $d + b(u,v,d)$ with a weight $p(d|u,v)$. The resulting continuous disparity estimate $D(u,v)$ at $(u,v)$ is the \textbf{mode} of $\tilde{p}(d'|u, v)$.

Our network design with a sub-network for offset prediction is reminiscent of G-RMI pose estimator~\cite{papandreou2017towards} and one-stage 2D object detectors~\cite{liu2016ssd,redmon2016you,lin2017focal}. The former
predicts the heatmaps (at fixed locations) and offsets for each keypoint; the latter parameterizes the predicted bounding box coordinates by the anchor box location plus the predicted offset. One may also interpret our approach as a coarse-to-fine depth prediction, first picking the bin centered around $\argmax_{d\in\mathcal{D}}p(d|u,v)$ and then locally adjusting it by an offset.

In our implementation, the sub-network $b(u,v,d)$ shares its feature and computation with $S_\text{disp}(u,v,d)$ except for the last block of  fully-connected or convolutional layers.

\subsection{Learning with Wasserstein distances}
\label{ssec:LWWD}
We propose to train our disparity network such that the mixture of Dirac delta functions (\autoref{e_mix_Dirac}) is directly learned to match the ground truth distribution. Concretely,
we represent the distribution of ground truth disparity at a pixel $(u,v)$, $p^\star(d'|u,v)$, as a Dirac delta function centered at the ground truth disparity  $d^\star=D^\star(u, v)$: $p^\star(d'|u,v) = \delta(d'-d^\star)$. We then employ a learning objective to minimize the divergence (distance) between $\tilde{p}(d'|u,v)$ and $p^\star(d'|u,v)$. There are many popular divergence measures between distributions, such as Kullback-Leibler divergence, Jensen-Shannon divergence, total Variation, the Wasserstein distance, etc. In this paper, we choose the Wasserstein distance for one particular reason: $\tilde{p}(d'|u,v)$ and $p^\star(d'|u,v)$ may not have any common supports.

The Wasserstein-$p$ distance between two distributions $\mu$, $\nu$ over a metric space $(X, d)$ is defined as
\begin{align}
W_{p}(\mu, \nu)=\left(\inf_{\gamma \in \Gamma(\mu, \nu)} \mathop{\mathbb{E}_\gamma} d(x, y)^{p} \right)^{1 / p}, \label{e_WD}
\end{align}
where $\Gamma(\mu, \nu)$ denotes the set of all the joint distributions $\gamma(x, y)$ whose marginal distributions  $\gamma(x)$ and $\gamma(y)$ are exactly $\mu$ and $\nu$, respectively. Intuitively, $\gamma(x, y)$ indicates how much ``mass'' to be transported from $x$ to $y$ in order to transform the distribution $\mu$ to $\nu$.

Estimating the Wasserstein distance is usually non-trivial and requires solving a linear programming problem. One particular exception is when $\mu$ and $\nu$ are both distributions of one-dimensional variables, which is the case for our distribution over disparity values\footnote{For dealing with disparity or depth values at a pixel, our metric space naturally becomes $\mathcal{R}^1$.}.
Specifically, when $\nu$ is a Dirac delta function whose support is located at $y^\star$, the Wasserstein-$p$ distance can be simplified as
\begin{align}
W_p(\mu, \nu) = (\mathop{\mathbb{E}_\mu}\mathop{\mathbb{E}_\nu} \left\| x - y \right\|^p)^{1 / p}  =(\mathop{\mathbb{E}_\mu} \left\| x - y^\star \right\|^p)^{1 / p}.\label{e_WD_Dirac}
\end{align}
By plugging $\tilde{p}(d'|u,v)$ and $p^\star(d'|u,v)$ into $\mu$ and $\nu$ respectively, we obtain
\begin{align}
W_p(\tilde{p}, p^\star) =(\mathop{\mathbb{E}_{\tilde{p}}} \left\| d' - d^\star \right\|^p)^{1 / p} & =\left(\sum_{d\in\mathcal{D}} p(d|u,v) \left\| d + b(u,v,d) - d^\star \right\|^p\right)^{1 / p} \label{e_loss_WD}\\
& =\left(\sum_{d\in\mathcal{D}} \softmax(-{\color{red}S_\text{disp}(u,v,d)}) \left\| d + {\color{blue}b(u,v,d)} - d^\star \right\|^p\right)^{1 / p}, \nonumber
\end{align}
based on which we can learn the conventional disparity network ({\color{red}red}) and the additional offset sub-network ({\color{blue}blue}) jointly (i.e.\xspace, by minimizing~\autoref{e_loss_WD}). We focus on $W_1$ and $W_2^2$ distances.

\subsection{Extension: learning with multi-modal ground truths}
\label{sec:mm-ext}

One particular advantage of learning to match the distributions is the capability of allowing multiple ground truth values (i.e.\xspace, a multi-modal ground truth distribution) at a single pixel location.
Denote $\mathcal{D}^\star$ as the set of ground truth disparity values at a pixel $(u, v)$, the ground truth distribution becomes
\begin{align}
{p}^\star(d'|u, v) = \sum_{d^\star\in\mathcal{D}^\star} \frac{1}{|\mathcal{D}^\star|} \delta(d' - d^\star).
\end{align}
Since ${p}^\star(d'|u, v)$ is not a Dirac delta function, we can no longer apply \autoref{e_WD_Dirac} but the following equation for comparing two one-dimensional distributions \cite{panaretos2020invitation,Wasserman20,ramdas2017wasserstein}
\begin{align}
W_{p}(\tilde{p}, p^\star) =\left(\int_{0}^{1}\left|\tilde{P}^{-1}(x)-{P^\star}^{-1}(x)\right|^{p}\text{d}x\right)^{1 / p}, \label{e_WD_one}
\end{align}
where $\tilde{P}$ and $P^\star$ are the cumulative distribution functions (CDFs) of $\tilde{p}$ and $p^\star$, respectively. For the case $p=1$, we can rewrite \autoref{e_WD_one} as \cite{thorpeintroduction}
\begin{align}
W_{1}(\tilde{p}, p^\star)=\int_\R\left|\tilde{P}(d')-P^\star(d')\right|\text{d}d'. \label{e_WD_one_one}
\end{align}
We note that, both \autoref{e_WD_one} and \autoref{e_WD_one_one} can be computed efficiently.

While existing datasets do not provide multi-modal ground truths directly, we investigate the following procedure to construct them. For each pixel, we consider a $k\times k$ neighborhood and create a multi-modal distribution by setting the center-pixel disparity with a weight $\alpha$ and the remaining ones each with $\frac{1-\alpha}{k\times k-1}$. We set $k=3$ and $\alpha=0.8$ in the experiment. Our empirical study shows that using a multi-modal ground truth leads to a much faster model convergence.

\subsection{Comparisons to related work}
\citet{kendall2017end} discussed the use of means or modes. They employed pre-scaling to sharpen the predicted probability, which might resolve the multi-modal issue but makes the prediction concentrate on discrete disparity values. In contrast, we do not prevent predicting a multi-modal distribution, especially for pixels whose disparities are inherently multi-modal. We output the mode (after an offset), which is what \citet{kendall2017end} hoped to achieve.
We note that 3D convolutions can smooth the estimation but cannot guarantee uni-modal distributions.

Compared to G-RMI pose estimator and one-stage 2D object detectors mentioned in~\autoref{ssec:CDN}, our work learns the two (sub-)networks jointly using a single objective function rather than a combination of two separated ones. See the supplementary material for more comparisons. \citet{liu2019conservative} propose to use the Wasserstein loss for pose estimation to characterize the inter-class correlations; however, they do not predict offsets for pre-defined discrete pose labels. Our work is also related to~\cite{bellemare2017distributional}, in which the authors propose to learn the value distribution, instead of the expected value, using the Wasserstein loss for reinforcement learning.

\section{Background}
\label{sec: related}

Stereo techniques rely on two cameras oriented parallel and translated horizontally relative to each other~\cite{yamaguchi2014efficient,zbontar2016stereo}.
In this setting, for a pixel $(u,v)$ in one image, the corresponding pixel in the second image is constrained to be at $(u + D(u,v), v)$, where $D(u,v)$ is called the \emph{disparity} of the pixel.
The disparity is inversely proportional to the \emph{depth} $Z(u,v)$ : $D(u,v) = \frac{f \times b}{Z(u,v)}$, where $b$ is the translation between the cameras (called the \emph{baseline}) and $f$ is the focal length of the cameras. Stereo depth estimation techniques typically first estimate disparity in units of pixels and then exploit the reciprocal relationship to approximate depth.
The basic approach is to compare pixels $(u,v)$ in the left image $I_l$ with pixels $(u,v+d)$ in the right image $I_r$ for different values of $d$, and find the best match.
Since pixel coordinates are constrained to be integers, $d$ is constrained to be an integer as well.
The estimated disparity is thus an integer, forcing the estimated depth to be one of a few discrete values.

Instead of producing a single integer-valued disparity value, modern pipelines produce a \emph{distribution} over these possible disparities~\cite{kendall2017end,chang2018pyramid}.
They do this by constructing a 4D \emph{disparity feature volume}, $C_\text{disp}$, in which $C_\text{disp}(u,v,d,:)$ is a feature vector that captures the difference in appearance between $I_l(u,v)$ and $I_r(u,v + d)$.
This feature vector can be, for instance, the concatenation of the feature vectors of the two pixels, in turn obtained by running a convolutional network on each image.
The disparity feature volume is then passed through a series of 3D convolutional layers, culminating in a cost for each disparity value $d$ for each pixel, $S_\text{disp}(u,v,d)$~\cite{chang2018pyramid}.
By taking $\softmax$ along the disparity dimension, one can turn $S_\text{disp}(u,v,d)$ into a probability distribution~\cite{luo2016efficient}.
Because we only consider integral disparity values, this distribution is a categorical distribution over the  possible disparity values (e.g.\xspace, $d\in\{0,\cdots,191\}$).
One can then obtain the disparity $D(u,v)$, for example, by $\argmax_d \softmax(-S_\text{disp}(u,v,d))$. However, in order to obtain continuous disparity estimates beyond integer-valued disparities, \cite{chang2018pyramid,wang2018anytime,kendall2017end,zhang2019ga} apply the following weighted combination (i.e.\xspace, mean),
\begin{align}
D(u, v) = \sum_d \softmax(-S_\text{disp}(u,v,d))\times d. \label{eq_disp_pred}
\end{align}
The whole neural network can be learned end-to-end, including the image feature extractor and 3D convolution kernels, to minimize the disparity error (on one image)
\begin{align}
\sum_{(u,v)\in \mathcal{A}} \ell(D(u,v) - D^\star(u,v)), \label{eq_disp_loss}
\end{align}
where $\ell$ is the smooth L1 loss, $D^\star$ is the ground truth map, and $\mathcal{A}$ contains pixels with ground truths.

Recently, \cite{you2019pseudo} argue that learning with \autoref{eq_disp_loss} may over-emphasize nearby depths, and accordingly propose to learn the network directly to minimize the depth loss. Specifically, they constructed depth cost volume  $S_\text{depth}(u,v,z)$, rather than $S_\text{disp}(u,v,d)$, and predicted the continuous depth by \begin{align}
Z(u, v) = \sum_z \softmax(-S_\text{depth}(u,v,z))\times z. \label{eq_depth_pred}
\end{align}
The entire network is learned to minimize the distance to the ground truth depth map $Z$
\begin{align}
\sum_{(u,v)\in \mathcal{A}} \ell(Z(u,v) - Z^\star(u,v)). \label{eq_depth_loss}
\end{align}
In this paper, we argue that the design choices to output continuous values (\autoref{eq_disp_pred} and \autoref{eq_depth_pred}) can be harmful to pixels in ambiguous regions, and the objective functions for learning the networks (\autoref{eq_disp_loss} and \autoref{eq_depth_loss}) do not directly match the predicted distribution to the true one. The most similar work to ours is~\cite{zhang2020adaptive}, which learns the network with a distribution matching loss on $\softmax(-S_\text{disp}(u,v,d))$; however, they still need to apply~\autoref{eq_disp_pred} to obtain continuous estimates. \citet{luo2016efficient} also learned the network by distribution matching, but applied post-processing (e.g.\xspace, semi global block matching) to obtain continuous estimates.

\paragraph{Stereo-based 3D object detection.} 3D object detection has attracted significant attention recently, especially for the application of self-driving cars~\cite{geiger2012we,geiger2013vision,argoverse,nuscenes2019,waymo_open_dataset,lyft2019}. While many algorithms rely on the expensive LiDAR sensor as input~\cite{shi2019pointrcnn,lang2019pointpillars,yang2018pixor,qi2018frustum,ku2018joint}, several recent papers have shown promising accuracy using the much cheaper stereo images~\cite{li2019gs3d,xu2018multi,chen2020dsgn,li2019stereo,konigshofrealtime,xu2020zoomnet,pon2019object}. One particular framework is Pseudo-LiDAR~\cite{pseudoLiDAR,you2019pseudo,qian2020end}, which converts stereo depth estimates into a 3D point cloud that can be inputted to any existing LiDAR-based detector, achieving the state-of-the-art results.

\section{Conclusion}
\label{sec: disc}

\begin{wrapfigure}{R}{0.45\textwidth}
    \vskip -50pt
    \centering
	\tabcolsep 1.5pt
	\includegraphics[width=\linewidth]{figures/pip7_2.pdf}
	\vskip-5pt
	\caption{\small \textbf{Qualitative results on disparity.} The top, middle, and bottom images are the left image, the result of \method{GANet}\xspace Deep, and the result of \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep, together with the foreground 3PE.
	} \label{fig:qualitative}
	\vskip-20pt
\end{wrapfigure}

In this paper we have introduced a new output representation, model architecture and loss function for depth/disparity estimation that can faithfully produce real-valued estimates of depth/disparity.
We have shown that results not just in more accurate depth estimates, but also significant improvement in downstream tasks like object detection.
Finally, because we explicitly output and optimize a distribution over depths, our approach can naturally take into account \emph{uncertainty and multimodality} in the ground truth.
More generally, our results suggest that removing suboptimalities in how we represent and optimize 3D information can have a large impact on a multitude of vision tasks.

\section{Experiments}
\label{sec: exp}

\subsection{Datasets and metrics}
\paragraph{Datasets.} We evaluate our method on two challenging stereo benchmark datasets, i.e., Scene Flow~\cite{mayer2016large} and KITTI 2015~\cite{menze2015object}, and on a 3D object detection
benchmark KITTI 3D~\cite{geiger2013vision,geiger2012we}.

\paragraph{1) Scene Flow~\cite{mayer2016large}.}
Scene Flow is a large synthetic dataset containing 35,454 training image pairs and 4,370 testing image pairs, where the ground truth disparity maps are densely provided, which is large enough for directly training deep neural networks.

\paragraph{2) KITTI 2015~\cite{menze2015object}.}
KITTI 2015 is a  real-world dataset with street scenes captured from a driving car. KITTI 2015 contains 200 training stereo image pairs with sparse ground truth disparities obtained using LiDAR, and 200 testing image pairs with ground truth disparities held by evaluation server for submission evaluation only. Its small size makes it a challenging dataset.

\paragraph{3) KITTI 3D~\cite{geiger2013vision,geiger2012we}.}
KITTI 3D contains 7,481 (pairs of) images for training and 7,518 (pairs of) images for testing. We follow the
same training and validation splits as suggested by~\citet{chen20153d}, containing 3,712 and 3,769 images, respectively. For each image, KITTI provides the corresponding Velodyne LiDAR point cloud (for sparse depth ground truths), camera calibration matrices, and 3D bounding box annotations. We evaluate our approach by plugging it into existing stereo-based 3D object detectors~\cite{you2019pseudo,pseudoLiDAR,chen2020dsgn}, which all require stereo depth estimation as a key component.

\paragraph{Metrics.} We evaluate our methods on three tasks: stereo disparity estimation, stereo depth estimation, and 3D object detection. We apply the corresponding standard metrics listed as follows.

\textbf{1) stereo disparity}, we use two standard metrics: End-Point-Error (EPE), i.e.\xspace, the average difference of the predicted disparities and their true
ones, and $k$-Pixel Threshold Error (PE), i.e.\xspace, the percentage of pixels for which the predicted disparity is off the ground truth by more than $k$ pixels. We use the 1-pixel and 3-pixel threshold errors, denoted as 1PE and 3PE. PE is robust to outliers with large disparity errors, while EPE measures errors to sub-pixel level.

\textbf{2) stereo depth.} We use the Root Mean Square Error (RMSE) $\sqrt{\frac{1}{|\mathcal{A}|}\Sigma_{(u,v)\in \mathcal{A}}|{z}(u,v)-z^\star(u,v)|^2 }$ and Absolute Relative Error (ABSR) $\frac{1}{|\mathcal{A}|}\Sigma_{(u,v)\in \mathcal{A}}\frac{|{z}(u,v)-z^\star (u,v)|}{z^\star(u,v)}$, where $\mathcal{A}$ denotes all the pixels having ground truths, and $z$ and $z^\star$ are estimated depth and ground truth depth respectively.

\textbf{3) 3D object detection.} We focus on 3D and bird's-eye-view (BEV) localization and report the results on the official leader board and the validation set. Specifically, we focus on the "car" category, following~\cite{chen2017multi, xu2018pointfusion}. We report the average precision (AP) at IoU thresholds 0.5 and 0.7. We denote AP for the 3D and BEV tasks by AP$_\text{3D}$\xspace and AP$_\text{BEV}$\xspace, respectively. The benchmark defines for each category three cases --- easy, moderate, and
hard -- according to the bounding box height and occlusion and truncation. In general, the easy cases correspond to cars within 30 meters of the ego-car distance.

\subsection{Implementation details}
\label{ssec:ID}
We mainly use the Wasserstein-1 distance (i.e.\xspace, $W_1$ loss) for training our \textbf{\method{CDN}}\xspace model. We compare $W_1$ and $W_2^2$ losses in the supplementary material.

\paragraph{Stereo disparity.}
We apply our \textbf{continuous disparity network (\textbf{\method{CDN}}\xspace)} architecture to PSMNet~\cite{chang2018pyramid} and GANet~\cite{zhang2019ga}, namely \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace and \textbf{\method{CDN}}\xspace-\method{GANet}\xspace.
To keep a fair comparison, we train the models with their default settings.
For Scene Flow, the models are trained from
scratch with a constant learning rate of 0.001 for 10 epochs. For KITTI 2015, the models pre-trained on Scene Flow are fine-tuned following the default strategy of the vanilla models.
We consider disparities in the range of $[0, 191]$ for both datasets.  We use a uniform grid of bin size $2$ pixels to create the categorical distribution (cf.~\autoref{eq_cat}).
We show the effect of bin sizes in the supplementary material.

\paragraph{Stereo depth.}
We apply \textbf{\method{CDN}}\xspace to the \method{SDN}\xspace architecture \cite{you2019pseudo}, namely \textbf{\method{CDN}}\xspace-\method{SDN}\xspace.
We follow the training procedure in \cite{you2019pseudo}. We consider depths in the range of $[0\text{m}, 80\text{m}]$.  We use a uniform grid of bin size $1$m to create the categorical distribution.

\paragraph{The offset sub-network.}
We implement $b(u,v,d)$ with a Conv3D-Relu-Conv3D block. It takes the 4D cost volume, before the last fully-connected or convolutional block of $S_\text{disp}(u,v,d)$, as the input. We predict a single offset $b(u,v,d)\in[0, s]$ for each integral disparity value $d$, where $s$ is the bin size. We achieve this by clipping. The sub-network has 30K parameters, only $0.3\%$ w.r.t. \method{PSMNet}\xspace~\cite{chang2018pyramid}. For stereo depth, we implement $b(u,v,z)\in[0, s]$ in the same way for each integral depth value $z$.

\paragraph{Stereo 3D object detection.}
We apply \textbf{\method{CDN}}\xspace-\method{SDN}\xspace to \method{pseudo-LiDAR}\xspace++ \cite{you2019pseudo}, which uses \method{SDN}\xspace to estimate depth. We fine-tune the \textbf{\method{CDN}}\xspace-\method{SDN}\xspace model pre-trained on Scene Flow on KITTI 3D dataset, followed by using an 3D object detector, here \method{P-RCNN}\xspace~\cite{shi2019pointrcnn}, to detect 3D bounding boxes of cars. %
We also apply \textbf{\method{CDN}}\xspace to \method{DSGN}\xspace~\cite{chen2020dsgn}, the state-of-the-art stereo-based 3D object detector. \method{DSGN}\xspace uses as a backbone depth estimator based on \method{PSMNet}\xspace and we replace it with our \textbf{\method{CDN}}\xspace version.

\paragraph{Multi-modal ground truths.}
As mentioned in \autoref{sec:mm-ext}, we create multi-modal ground truths for a pixel by considering a patch in its $k\times k$ neighborhood. We give the center-pixel disparity a weight $\alpha = 0.8$, and the remaining ones an equal weight such that the total sums to 1. In this case, we use \autoref{e_WD_one_one} as the loss function. We implement a differentiable loss module in Pytorch that can be applied to a batch of image tensors. Please see the supplementary material for more details.

\begin{table}[]
	\centering
	\small
	\caption{\small \textbf{Disparity results}. We report results on Scene Flow and KITTI 2015. For Scene Flow, end point errors (EPE) and the 1-pixel and 3-pixel threshold error rates (1PE, 3PE) are reported. For KITTI 2015 we report the standard metrics (using 3PE) for both Non-occluded and All pixels regions. Methods based on \textbf{\method{CDN}}\xspace are highlighted in {\color{blue} blue}. Lower is better. The best result per column in in bold. \emph{Since the baselines are mostly trained with uni-modal ground truths, we only show \textbf{\method{CDN}}\xspace with the same ground truths here for a fair comparison.}} \label{tbMain}
	\begin{tabular}{=l|+c|+c|+c|+c|+c|+c|+c}  & \multicolumn{3}{c|}{Scene Flow} & \multicolumn{4}{c}{KITTI 2015} \\ \cline{2-8}
	 &  & & & \multicolumn{2}{c|}{Non Occlusion 3PE} & \multicolumn{2}{c}{All Areas 3PE}\\  \cline{5-8}
		\multicolumn{1}{c|}{Method} & EPE & 1PE & 3PE & Foreground & All & Foreground & All \\ \hline
		MC-CNN~\cite{zbontar2016stereo} &  3.79 & - & - & 7.64 & 3.33 & 8.88 & 3.89 \\
		GC-Net~\cite{kendall2017end} & 2.51 & 16.9 & 9.34 & 5.58 & 2.61 & 6.16 & 2.87 \\
        PSMNet~\cite{chang2018pyramid} & 1.09 & 12.1 & 4.56 & 4.31 & 2.14 & 4.62 & 2.32 \\
        SegStereo~\cite{yang2018segstereo} & 1.45 & - & - & 3.70 & 2.08 & 4.07 & 2.25\\
        GwcNet-g~\cite{guo2019group} & 0.77 & 8.0 & 3.30 & 3.49 & 1.92 & 3.93 & 2.11\\
        HD\textsuperscript{3}-Stereo~\cite{Yin_2019_CVPR} & 1.08 & - & - & 3.43 & 1.87 & 3.63 & 2.02\\
        GANet~\cite{zhang2019ga} & 0.84 & 9.9  & - & 3.37 & 1.73 & 3.82 & 1.93 \\
        AcfNet~\cite{zhang2020adaptive} & 0.87 & - & 4.31 & 3.49 & 1.72 & 3.80 & 1.89 \\
        Stereo Expansion~\cite{yangupgrading} & - & -&- & 3.11 & 1.63 & 3.46 & 1.81 \\
        GANet Deep~\cite{zhang2019ga} & 0.78 & 8.7 & - & 3.11 & \textbf{1.63} & 3.46 & \textbf{1.81} \\ \hline
        \rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-PSMNet    & 0.98 & 9.1 & 3.99 & 4.01 & 2.12 & 4.34 & 2.29  \\
		\rowstyle{\color{blue}}
		\rowstyle{\color{blue}} \textbf{\method{CDN}}\xspace-GANet Deep & \textbf{0.70} & \textbf{7.7} & \textbf{2.98} & \textbf{2.79} & 1.72 & \textbf{3.20} & 1.92   \\
		\hline
	\end{tabular}
	\label{tbl:disp}
	\vskip-10pt
\end{table}

\subsection{Main results}
\label{ssec:ER}

\paragraph{Disparity estimation.} \autoref{tbl:disp} summarizes the results on disparity estimation. \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep\footnote{We apply the \method{GANet}\xspace Deep model introduced in the released code of~\cite{zhang2019ga}, available at~\url{https://github.com/feihuzhang/GANet}. The main architectures of \method{GANet}\xspace Deep and \method{GANet}\xspace are the same, while the former has some more 2D and 3D convolutional layers.} achieves the lowest error at all three metrics on Scene Flow. It reduces the error for \method{GANet}\xspace Deep by $1.0$ 1PE and $0.08$ EPE, both are significant. We see a similar gain for \method{PSMNet}\xspace: \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace reduces EPE by $0.09$, demonstrating the general applicability of our approach to existing networks.

On KITTI 2015, \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep obtains the lowest error on the \emph{foreground} pixels and performs comparably to other methods on all the pixels\footnote{There are two possible reasons that \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep does not outperform \method{GANet}\xspace Deep on all the pixels. First, \textbf{\method{CDN}}\xspace overly focuses on foreground pixels. Second, we used the same hyper-parameters as the original \method{GANet}\xspace without specific tuning for \textbf{\method{CDN}}\xspace. We note that the ratio of foreground/background pixels is $\sim0.15/0.85$; the degradation by \textbf{\method{CDN}}\xspace on the background is $\sim0.16$ 3PE, smaller than the gain on foreground.}. We see a similar gain by \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace over \method{PSMNet}\xspace on the foreground, which is quite surprising, as we do not specifically \emph{re-weight} the loss function towards foreground pixels. Since \textbf{\method{CDN}}\xspace has advantages on pixels whose disparity is ambiguous and hard to estimate correctly (e.g.\xspace, due to multi-modal distributions), the fact that foreground pixels have a higher error and \textbf{\method{CDN}}\xspace can effectively reduce it suggests that those challenging pixels are mostly in the foreground.
As will be seen in 3D object detection, the improvement by \textbf{\method{CDN}}\xspace on foreground pixels translates to a higher accuracy on localizing objects.

\begin{table}[]
\small
	\centering
	\caption{\small \textbf{3D object detection results on the KITTI leader board.} We report AP$_\text{BEV}$\xspace and AP$_\text{3D}$\xspace (in \%) of the \textbf{car} category at IoU$=0.7$. Methods with \textbf{\method{CDN}}\xspace are in {\color{blue} blue}. The best result of each column is in bold font.}  %
	\label{tbl:3d_object_detection}
	\begin{tabular}{=l|+c|+c|+c|+c|+c|+c}
		 & \multicolumn{3}{c|}{BEV Detection AP (AP$_\text{BEV}$\xspace)} & \multicolumn{3}{c}{3D Detection AP (AP$_\text{3D}$\xspace)} \\ \cline{2-7}
		\multicolumn{1}{c|}{Method} & Easy & Moderate & Hard & Easy & Moderate & Hard \\ \hline
		\method{S-RCNN}\xspace~\cite{li2019stereo} & 61.9 & 41.3 & 33.4 & 47.6 & 30.2 & 23.7 \\
		\method{OC-Stereo}\xspace~\cite{pon2019object} & 68.9 & 51.5 & 43.0 & 55.2 & 37.6 & 30.3\\
		\method{Disp R-CNN}\xspace~\cite{sun2020disprcnn} & 74.1 & 52.4 & 43.8 & 59.6 & 39.4 & 32.0\\ \hline
		\method{pseudo-LiDAR}\xspace~\cite{pseudoLiDAR}    & 67.3  & 45.0 & 38.4 & 54.5 & 34.1 & 28.3  \\
		\method{pseudo-LiDAR}\xspace++~\cite{you2019pseudo}    & 78.3 & 58.0  & 51.3 & 61.1 & 42.4 & 37.0\\
		\method{pseudo-LiDAR}\xspace E2E~\cite{qian2020end}   & 79.6 & 58.8 & 52.1 & 64.8 & 43.9 & 38.1 \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{pseudo-LiDAR}\xspace++    & 81.3 & 61.0 & 52.8 & 64.3 & 44.9 & 38.1 \\ \hline
 		\method{DSGN}\xspace~\cite{chen2020dsgn} & 82.9 & 65.0 & 56.6 & 73.5 & 52.2 & 45.1\\
 		\rowstyle{\color{blue}}
        \textbf{\method{CDN}}\xspace-\method{DSGN}\xspace & \textbf{83.3} & \textbf{66.2} & \textbf{57.7} & \textbf{74.5} & \textbf{54.2} & \textbf{46.4}\\ \hline
	\end{tabular}
	\vskip-10pt
\end{table}

\begin{wrapfigure}{R}{0.35\textwidth}
	\vspace{-13pt}
	\includegraphics[width=\linewidth]{figures/mm.pdf}
	\vskip-5pt
	\caption{\small \textbf{MM training.} We show the EPE and 3PE disparity errors on Scene Flow test set using \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace, w/ or w/o MM training. MM training leads to faster convergence.} \label{fig:converge}
	\vskip -5pt
\end{wrapfigure}

\paragraph{3D object detection.}
\autoref{tbl:3d_object_detection} summarizes the results on the test set of KITTI 3D. Our \textbf{\method{CDN}}\xspace consistently improves the two mainstream approaches, namely, \method{DSGN}\xspace and \method{pseudo-LiDAR}\xspace. For \method{pseudo-LiDAR}\xspace, we achieve a $2.5\%$/$3.0\%$ gain on AP$_\text{3D}$\xspace/AP$_\text{BEV}$\xspace Moderate (the standard metric on the leader board) against \method{pseudo-LiDAR}\xspace++: the only difference is that we replace \method{SDN}\xspace by our \textbf{\method{CDN}}\xspace-\method{SDN}\xspace to have better depth estimates. Our approach even outperforms \method{pseudo-LiDAR}\xspace E2E, which fine-tunes the depth network specifically for object detection. We argue that our approach, which can \emph{automatically} focus on the foregrounds, may have a similar effect as end-to-end training with object detection losses.
For \method{DSGN}\xspace, plugging our \textbf{\method{CDN}}\xspace-\method{SDN}\xspace leads to a notable $2\%$ gain at AP$_\text{3D}$\xspace, attaining the highest entry of stereo-based 3D detection accuracy on the KITTI leader board.

\iffalse
\begin{table}[t]
\small
	\begin{minipage}{.46\textwidth}
		\small
		\caption{\small \textbf{Disparity multi-modal results.} We report the EPE, 1PE and 3PE on Scene Flow. Methods with \textbf{\method{CDN}}\xspace are highlighted in {\color{blue} blue}. The best result of each column is in bold font.}  %
	\label{tbl:disp_mm}
	\begin{tabular}{=l|+c|+c|+c}
		Method & EPE & 1PE & 3PE \\ \hline
        \method{PSMNet}\xspace & 1.09 & 12.1 & 4.56 \\
		\rowstyle{\color{blue}}
		CDN-\method{PSMNet}\xspace    & 0.98 & 9.1 & 3.99   \\
		\rowstyle{\color{blue}}
		CDN-\method{PSMNet}\xspace MM & \textbf{0.96} & \textbf{9.0} & \textbf{3.96}  \\ \hline
	\end{tabular}
    \end{minipage}
\hfill
    \begin{minipage}{.46\textwidth}
		\small
		\caption{\small \textbf{Depth multi-modal results.} We report the RMSE and ABSR errors on Scene Flow. The best result of each column is in bold font.}  %
	\label{tbl:depth}
	\begin{tabular}{=l|+c|+c}
		Method & RMSE (m) & ABSR \\ \hline
		\method{SDN}\xspace  & 2.05 & 0.039  \\
		\rowstyle{\color{blue}}
	    \textbf{\method{CDN}}\xspace-\method{SDN}\xspace    & 1.81 & 0.030   \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace MM & \textbf{1.80} & \textbf{0.028}     \\ \hline
	\end{tabular}
    \end{minipage}
    \vskip -10pt
\end{table}
\fi

\begin{table}[t]
\small
\caption{\small \textbf{Disparity multi-modal results.} We report the EPE, 1PE and 3PE on Scene Flow. Methods with \textbf{\method{CDN}}\xspace are highlighted in {\color{blue} blue}. The best result of each column is in bold font.}
\label{tbl:disp_mm}

\begin{minipage}{.48\textwidth}
		\small
	\begin{tabular}{=l|+c|+c|+c}
		Method & EPE & 1PE & 3PE \\ \hline
        \method{PSMNet}\xspace~\cite{chang2018pyramid} & 1.09 & 12.1 & 4.56 \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace    & 0.98 & 9.1 & 3.99   \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace MM & \textbf{0.96} & \textbf{9.0} & \textbf{3.96}  \\ \hline
	\end{tabular}
    \end{minipage}
\hfill
    \begin{minipage}{.48\textwidth}
		\small  %
	\begin{tabular}{=l|+c|+c|+c}
		Method & EPE & 1PE & 3PE \\ \hline
        \method{GANet}\xspace Deep~\cite{zhang2019ga} & 0.78 & 8.7 & - \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep   & 0.70 & 7.7 & 2.98   \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep MM & \textbf{0.68} & \textbf{7.7} & \textbf{2.97}  \\ \hline
	\end{tabular}
    \end{minipage}
    \vskip -10pt
\end{table}

\begin{table}[t]
\small
\begin{minipage}{.48\textwidth}
	\small
	\caption{\small \textbf{Depth multi-modal results.} We report the RMSE and ABSR errors on Scene Flow.
	The best result of each column is in bold font.}
	\label{tbl:depth}
	\begin{tabular}{=l|+c|+c}
		Method & RMSE (m) & ABSR \\ \hline
		\method{SDN}\xspace~\cite{you2019pseudo}  & 2.05 & 0.039  \\
		\rowstyle{\color{blue}}
	    \textbf{\method{CDN}}\xspace-\method{SDN}\xspace    & 1.81 & 0.030   \\
		\rowstyle{\color{blue}}
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace MM & \textbf{1.80} & \textbf{0.028}     \\ \hline
\end{tabular}
    \end{minipage}
\hfill
    \begin{minipage}{.48\textwidth}
		\small
	\caption{\small \textbf{Ambiguous regions (object boundaries).}  We report the disparity error on Scene Flow. The best result of each column is in bold font.}
	\label{tbl:eval_ambigious}
	\begin{tabular}{=l|+c|+c|+c}
		Method & EPE & 1PE & 3PE \\ \hline
		PSMNet~\cite{chang2018pyramid} & 3.10 & 20.1 & 11.33 \\\rowstyle{\color{blue}}
		CDN-PSMNet & 2.10 & 15.3 & 8.92 \\
		\rowstyle{\color{blue}}
		CDN-PSMNet MM & \textbf{2.08} & \textbf{13.2} & \textbf{8.65} \\
		\hline
	\end{tabular}
    \end{minipage}
    \vskip -10pt
\end{table}

\subsection{Analysis}
\label{ssec:ER}

\paragraph{Multi-modal (MM) ground truth.}
We investigate creating the multi-modal (MM) ground truths for training our models.  \autoref{tbl:disp_mm} and \autoref{tbl:depth} summarize the results on Scene Flow for disparity and depth estimation, respectively. MM training slightly reduces the errors. To better understand how MM ground truths affect network training, we plot the test accuracy along the training epochs in~\autoref{fig:converge}: \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace trained with MM ground truths converges much faster. We attribute this to the observations in~\cite{arpit2017closer}: a neural network tends to learn simple and \emph{clean} patterns first. We note that, for boundary pixels whose disparities are inherently multi-modal, uni-modal ground truths are indeed \emph{noisy} labels. A network thus tends to ignore these pixels in the early epochs. In contrast, MM ground truths provide \emph{clean} supervisions for these boundary pixels; the network thus can learn the patterns much faster. See the supplementary material for a visualization and further discussions.

\begin{wraptable}{R}{0.45\linewidth}
	\vskip -8pt
	\centering
	\small
	\tabcolsep 3pt
	\vskip -5pt
	\caption{\small \textbf{Ablation studies.} We report disparity error for \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace on Scene Flow. Methods without $W_1$ loss are learned with \emph{mean} regression.}  %
	\label{tbl:depth_abl}
	\begin{tabular}{=l|+c|+c|+c|+c|+c}
		Offsets & $W_1$ Loss & Output & EPE& 1PE & 3PE \\ \hline
		& & Mean & 1.09 & 12.1 & 4.56 \\
		\hspace{5pt} \checkmark & & Mean & 1.04 & 12.0 & 4.55\\
		& \checkmark & Mode & 1.20 & 10.5 & 4.21 \\
		\hspace{5pt} \checkmark & \checkmark & Mode &\textbf{0.98} & \textbf{9.1} & \textbf{3.99} \\ \hline
	\end{tabular}
\vskip -5pt
\end{wraptable}

\paragraph{Ablation studies.} We study different components of our approach in~\autoref{tbl:depth_abl}. Methods without $W_1$ loss use the regression loss for optimization (cf. \autoref{eq_disp_loss}) and output the mean. Methods with $W_1$ loss output the mode.
We see that, the offset sub-network alone can hardly improve the performance. Using $W_1$ distance alone reduces 1PE and 3PE errors, but not EPE, suggesting that it cannot produce sub-pixel disparity estimates\footnote{Using a bin size $s=2$ without offsets, the mode is restricted to integral values and EPE suffers.}. Only combining the offset sub-network and the $W_1$ loss produces consistent improvement over all three metrics.

\iffalse
\begin{wraptable}{R}{0.35\linewidth}
	\centering
	\small
	\vskip-12pt
	\tabcolsep 2pt
	\caption{\small \textbf{Ambiguous regions.}  We report the disparity error on Scene Flow.}
	\vskip-5pt
	\label{tbl:eval_ambigious}
	\begin{tabular}{=l|+c|+c|+c}
		Method & EPE & 1PE & 3PE \\ \hline
		PSMNet~\cite{chang2018pyramid} & 3.10 & 20.1 & 11.33 \\\rowstyle{\color{blue}}
		CDN-PSMNet & 2.10 & 15.3 & 8.92 \\
		\rowstyle{\color{blue}}
		CDN-PSMNet MM & 2.08 & 13.2 & 8.65 \\
		\hline
	\end{tabular}
	\vskip-5pt
\end{wraptable}
\fi

\paragraph{Disparity on boundaries.}
\autoref{tbl:eval_ambigious} shows the results: we obtain pixels on object boundaries using the OpenCV Canny edge detector with minVal/maxVal=100/200.
Both \textbf{\method{CDN}}\xspace and training with multi-modal ground truths reduce the error significantly.

\paragraph{Qualitative disparity results on KITTI.}
As shown in~\autoref{fig:qualitative}, our approach is able to estimate disparity accurately, especially along the object boundaries. Specifically, \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep maintains the straight bar shape (on the right), while \method{GANet}\xspace Deep blends it with the background sky due to the mean estimates.

\section{Introduction}
\label{sec: intro}

\begin{wrapfigure}{r}{0.5\textwidth}
		\vspace{-32pt}
		\includegraphics[width=\linewidth]{figures/teaser_new.pdf}
		\vspace{-10pt}
		\caption{\textbf{The effect of our continuous disparity network (CDN).} We show a person (green box) in front of a wall. The blue 3D points are obtained using PSMNet~\cite{chang2018pyramid}. The red points from our \textbf{CDN} model are much better aligned with the shape of the objects: they do not suffer the streaking artifacts near edges. Yellow points are from the ground truth LiDAR. (One floor square is 1m$\times$1m.) \label{fig:teaser}}
		\vskip -10pt
\end{wrapfigure}

Depth estimation from stereo images is a long-standing task in computer vision~\cite{scharstein2003high,papandreou2017towards}.
It is a key component of many downstream problems, ranging from 3D object detection in autonomous vehicles~\cite{pseudoLiDAR,you2019pseudo,chen2020dsgn,li2019stereo,qian2020end} to graphics applications such as novel view generation~\cite{yu2019depth,liu2018geometry}.
The importance of this task in practical applications has led to a flurry of recent research.
Convolutional networks have now superseded more classical techniques and led to significant improvements in accuracy~\cite{wang2018anytime,chang2018pyramid,mayer2016large,zhang2019ga}.

These techniques estimate depth by finding accurate pixel correspondences and estimating the \emph{disparity} between their $X$-coordinates, which is inversely proportional to depth.
Because pixels have integral coordinates, so does the estimated disparity --- causing even the resulting depth estimates to be discrete.
This introduces inaccuracy, as the ground truth disparity and depth are naturally real-valued. This discrepancy is typically addressed by predicting a \emph{categorical distribution} over a fixed set of discrete values, and then computing the \emph{expected} depth from this distribution, which can in theory be any arbitrary real value (within the range of the set)~\cite{wang2018anytime,chang2018pyramid,zhang2019ga,kendall2017end,you2019pseudo}.

In this paper, we argue that such a design choice may lead to inaccurate depth estimates, especially around object boundaries.
For example, in \autoref{fig:teaser} we show the pixels (back-projected into 3D using the depth estimates) along the boundary between a person in the foreground at $30$m depth and a wall in the background at $70$m depth. The predicted depth distribution of these border pixels is likely to be multi-modal, having two peaks around $30$ and $70$ meters. Simply taking the mean outputs a low probability value in between the two modes (e.g.\xspace, $50$m).
Such ``smoothed'' depth estimates can have a strong negative impact on subsequent 3D object detection, as they  ``smear'' the pedestrian around the edges towards the background (note the many blue points between the wall and the pedestrian). A bounding box including all these trailing points, far from the actual person, would strongly misrepresent the scene's geometry. What may further aggravate the problem is how the distribution is usually learned.
Existing approaches mostly learn the distribution via a regression loss: minimizing the distance between the mean value and the ground truth~\cite{kendall2017end,you2019pseudo}.
In other words, there is no direct supervision to teach the model to assign higher probabilities around the truth depth.

To address these issues, we propose a novel neural network architecture for stereo disparity estimation that is capable of outputting a distribution over \emph{arbitrary} disparity values, from which we can directly take the mode and bypass the mean.
As with existing work, our model predicts a probability for each disparity value in a pre-defined, discrete set.
Additionally, it predicts a real-valued \emph{offset} for each discrete value.
This is a simple architectural modification, but it has a profound impact.
With these offsets, the output is converted from a discrete categorical distribution to a \emph{continuous} distribution
 over disparity values: a  mixture of Dirac delta functions, centered at the pre-defined discrete values shifted by predicted offsets\footnote{Our work is reminiscent of G-RMI pose estimator~\cite{papandreou2017towards}, which predicts the heatmaps (at fixed locations) and offsets for each keypoint. Our work is also related to one-stage object detectors~\cite{redmon2016you,liu2016ssd,lin2017focal} that predict the class probabilities and box offsets for each anchor box.}.
This simple addition of predicted offsets allows us to use the mode as the prediction during inference, instead of the mean, guaranteeing that the predicted depth has a high estimated probability. \autoref{fig:illustration} illustrates our model, \textbf{continuous disparity network (CDN)}.

Next, we propose a novel loss function that provides a more informative objective during training. Concretely, we allow uni- or multi-modal ground truth depth distributions (obtained from nearby pixels) and represent them as (mixtures of) Dirac delta functions. The learning objective is then to minimize the divergence between the predicted and the ground truth distributions.
Noting that the two distributions might not have a common support, we apply the Wasserstein distance~\cite{villani2008optimal} to measure the divergence. While computing the exact Wasserstein distance of arbitrary distributions can be time-consuming, computing it for one-dimensional distributions (e.g.\xspace, distributions of one-dimensional disparity) enjoys efficient solutions, creating negligible training overhead.

Our proposed approach is both mathematically well-founded and practically extremely simple.
It is compatible with most existing stereo depth or disparity estimation approaches --- we only need to add an additional offset branch and replace the commonly used regression loss by the Wasserstein distance.
We validate our approach using multiple existing stereo networks~\cite{chang2018pyramid,you2019pseudo,zhang2019ga} on three tasks: stereo disparity estimation~\cite{mayer2016large}, stereo depth estimation~\cite{geiger2012we}, and 3D object detection~\cite{geiger2012we}. The last is a downstream task using stereo depth as the input to detect objects in 3D. We conduct comprehensive experiments and show that our algorithm leads to significant improvement in all three tasks.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/pipelinefig6.pdf}
\vskip-5pt
\caption{\small\textbf{Continuous disparity network (CDN).} We propose to predict a \emph{real-value} offset ({\color{dandelion}yellow arrows}) for each pre-defined discrete disparity value (e.g.\xspace, $\{1,2,3,4\}$), turning a categorical distribution ({\color{carnationpink}magenta bars}) to a continuous distribution ({\color{red}red bars}), from which we can output the mode disparity for accurate estimation.}
\label{fig:illustration}
\vskip-10pt
\end{figure}

\section*{Broader Impact}
The end results of this paper are improved depth and disparity estimation, particularly on foreground objects.
This is of use to self-driving cars, 3D reconstruction, and other robotics applications.
In particular, it has the potential to improve the \emph{safety} of these systems, as indicated by the increased 3D object detection performance.
Our approach can also easily be incorporated into other depth or disparity estimation algorithms for further improvement.

While our depth predictions are significantly better, any failure has important safety considerations, such as collisions and accidents.
Before deployment, appropriate safety thresholds must be cleared.

Our approach does not specifically leverage dataset biases, although being a machine learning approach, it is impacted as much as other machine learning techniques.

\section*{Acknowledgments}
This research is supported by grants from the National Science Foundation NSF (III-1618134, III-1526012, IIS-1149882, IIS-1724282, and TRIPODS-1740822, OAC-1934714), the Office of Naval Research DOD (N00014-17-1-2175), the Bill and Melinda Gates Foundation, and the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875). We are thankful for generous support by Zillow, SAP America Inc, AWS Cloud Credits for Research, Ohio Supercomputer Center, and Facebook.

\section{Implementation Details}
\label{supsec:detail}

\subsection{Learning with multi-modal ground truths}

For multi-modal ground truths, we cannot use Equation 8 of the main paper for optimization. Instead, we apply the loss in Equation 12, for $W_1$ distance.
This loss essentially computes the difference in areas between the CDFs of the two distributions. For mixtures of Dirac delta functions, it can be efficiently implemented by computing the accumulated difference between CDF histograms. It takes $\mathcal{O}(B\log B)$ for each pixel using sorting, where $B$ is the total number of supports of both distributions.
Our implementation is adapted from \emph{scipy.stats.wasserstein\_distance} and we modify it to be compatible with Pytorch tensors and use CUDA to parallelize the computation over all the pixels.

\subsection{Learning with the (approximated) KL divergence}
\label{supssec:kl_divergence}

The \emph{Kullback–Leibler} (KL) divergence
\begin{align}
KL(\mu || \nu) = \mathop{\mathbb{E_\mu}} \log (\mu|\nu)
\end{align}
between two distributions $\mu$ and $\nu$ requires them to have the same supports: i.e.\xspace, $\mu(d') = 0$ if $\nu(d') = 0$, for $KL(\mu || \nu)$ to be finite.

For our case, $\mu(d')=\delta(d'-d^\star)$ and $\nu(d') = \sum_{d\in\mathcal{D}} p(d|u,v) \delta(d' - (d + b(u,v,d)))$.
These two measures may have different supports. To make the KL divergence applicable, we can smooth $\nu$ to form a mixture of Laplace or Gaussian distributions.

For example, smoothing $\nu$ with a Laplace distribution, $
\text{Laplace}(0, \tau)=\frac{1}{2 \tau} \exp\left(-\frac{|d'|}{\tau}\right)$, we get
\begin{align}
\nu_\text{Lap}(d') = \sum_{d\in\mathcal{D}} p(d|u,v) \frac{1}{2 \tau} \exp\left(-\frac{| d' - (d + b(u,v,d))|}{\tau}\right).
\end{align}
With $\nu_\text{Lap}$, the KL divergence reduces to the following loss
\begin{align}
\ell(\mu, \nu_\text{Lap}) & = -\log\sum_{d\in\mathcal{D}} p(d|u,v) \frac{1}{2 \tau} \exp\left(-\frac{| d^\star - (d + b(u,v,d))|}{\tau}\right)\\
& \approx -\log p(\bar{d}|u,v) + \frac{1}{\tau} \left| d^\star - (\bar{d} + b(u,v,\bar{d})) \right|,\label{eq_Lap_loss}
\end{align}
where $\bar{d} = s\lfloor \frac{d^\star}{s} \rfloor\in\mathcal{D}$ is the grid disparity value of the bin the true disparity $d^\star$ belongs to.

Similarly, smoothing $\nu$ with a Gaussian distribution $\mathcal{N}(0, \sigma^{2})$, we get
\begin{align}
\nu_\text{Gau}(d') = \sum_{d\in\mathcal{D}} p(d|u,v) \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{\| d' - (d + b(u,v,d))\|_2^2}{2\sigma^2}\right).
\end{align}
With $\nu_\text{Gau}$, the KL divergence reduces to the following loss
\begin{align}
\ell(\mu, \nu_\text{Gau}) & = -\log\sum_{d\in\mathcal{D}} p(d|u,v) \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{\| d^\star - (d + b(u,v,d))\|_2^2}{2\sigma^2}\right)\\
& \approx -\log p(\bar{d}|u,v) + \frac{1}{2\sigma^2} \left\| d^\star - (\bar{d} + b(u,v,\bar{d})) \right\|_2^2,
\end{align}
where $\bar{d} = s\lfloor \frac{d^\star}{s} \rfloor\in\mathcal{D}$ is the grid disparity value of the bin the true disparity $d^\star$ belongs to.

These formulations reduce to the conventional classification loss plus offset regression loss, commonly used for keypoint estimation~\cite{zhou2019objects,papandreou2017towards} and one-stage 2D object detection~\cite{zhou2019objects,redmon2016you,liu2016ssd,lin2017focal}.

\section{Additional Discussions}
\label{supsec:disc}

\subsection{Multi-modal ground truths} There are three reasons why multi-modal ground truths would benefit disparity or depth estimation. First, pixels are discrete: a single pixel may capture different depths. Second, real datasets need to project signals from a depth sensor (e.g.\xspace, LiDAR) to a depth map. As pixels are discrete and the cameras and LiDAR might be placed differently, multiple LiDAR points of different depths may be projected to the same pixel. Third, for stereo estimation, pixels along boundaries or occluded regions cause ambiguity to the model; multi-modal ground truths offer better supervision for training, especially in early training epochs.

Conceptually, learning with multi-modal ground truths should notably improve results in Table 3 and Table 4 of the main paper. However, in evaluation, a majority of pixels are not on the object boundaries. Besides, we still evaluate using the (likely noisy) uni-modal ground truths. To further analyze these, we show in Table 6 of the main paper the disparity error calculated on object boundaries: learning with multi-modal ground truths leads to a significant improvement.

\subsection{Multi-task learning}
One way to mitigate stereo predictions at depth discontinuities is to jointly perform stereo estimation and other tasks such as semantic segmentation~\cite{jiao2018look,xu2018pad,jafari2017analyzing}, which can reason about object boundaries. The core idea is to leverage additional semantic labels to guide the model to resolve depth discontinuities (i.e.\xspace, predict uni-modal distributions). Our method, in contrast, does not prevent predicting multi-modal distributions along depth discontinuities, but changes the outputting rule (i.e.\xspace, $\argmin$ with a predicted offset).
Our method can also capture depth discontinuities within an object or an object class. In contrast, semantic segmentation labels overlapped objects of the same class by the same label and does not directly tell their boundaries.

\subsection{Offsets and distributions without common supports} While the \emph{learned} offsets may lead to common supports between the predicted and ground truth distributions, we have to first come up with a loss to \emph{learn} such offsets before common supports become possible.
Concretely, to learn $b(u,v,d)$ in Equation 9, we need a loss that can measure the divergence between $\tilde{p}$ and $p^\star$, which may not have common supports. We note that, this may occur even if the target distribution $p^\star$ is a Dirac delta function.
While the KL divergence or a regression loss may be applied to learn the offsets, they need to either smooth the distributions or carefully design the loss to learn both the distribution and the offset networks.
The Wasserstein distance offers a principled loss to learn the two networks jointly.

\section{Additional Results and Analysis}

\label{supsec:result_analysis}

\subsection{Ablation studies on different divergences}
\label{supsec:loss}
We show the ablation study on using different divergences between distributions in \autoref{tbl:loss_ablation}. For the KL divergence (\autoref{supssec:kl_divergence}), we use Laplace smoothing with $b=1$ (\autoref{eq_Lap_loss}). Our results show that the Wasserstein distance is a better choice than the KL divergence for comparing the predicted and the ground truth disparity (or depth) distributions. We also see that $W_2^2$ distance performs worse than $W_1$. We attribute this to outliers (i.e.\xspace, noisy disparity labels) in a dataset.

\begin{table}[]
		\small
		\centering
		\caption{\small \textbf{Comparison of different divergences (distances).} We report the RMSE and the ABSR error for depth estimation on Scene Flow. The best result of each column is in bold font.}  %
		\label{tbl:loss_ablation}
		\begin{tabular}{l|c|c|c}
		Method & Divergence &  RMSE (m) & ABSR \\ \hline
		\method{SDN}\xspace  & - & 2.05 & 0.04  \\
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace & KL    & 2.57 & 0.04   \\
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace & $W_1$    & \textbf{1.81} & \textbf{0.03}   \\
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace & $W_2^2$  & 1.91 & 0.05   \\ \hline
	\end{tabular}
\end{table}

\subsection{Effect of bin sizes}
\textbf{\method{CDN}}\xspace outputs modes and needs (a) the bin containing the truth disparity or depth to have the highest probability and (b) the offset to be accurate.
The bin size balances the difficulty of (a) and (b).
A smaller bin size makes (a) harder. A larger bin size makes (a) easier but makes (b) harder as the range of offsets gets larger. It is the only hyper-parameter to tune and only integral values are considered.

We show these effects of bin sizes on uniform grids, with disparities in the range of $[0, 191]$ for disparity estimation in \autoref{tbl:abl_bins}. For a bin size $s=1$, predicting the correct bin is harder.
For a bin size $s=4$, predicting the correct bin is easier, whereas predicting the correct offset becomes harder. We found $s=2$ to perform well in general.

\begin{table}[]
	\centering
	\small
	\caption{\small \textbf{Comparison of bin sizes.}  We report the disparity error on Scene Flow using \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace model.}
	\label{tbl:abl_bins}
	\begin{tabular}{=c|+c|+c|+c}
		Bin size & EPE & 1PE & 3PE \\ \hline
		1 & 1.22 & 13.9 & 4.33 \\
		2 & \textbf{0.98} & \textbf{9.1} & \textbf{3.99} \\
		4 & 1.52 & 26.1 & 4.17 \\
     \hline
	\end{tabular}
\end{table}

\begin{table}[t]
    \centering
	\caption{ \small \textbf{Ablation studies on the MM ground truths.} We conduct experiments using \textbf{\method{CDN}}\xspace-\method{SDN}\xspace MM on Scene Flow (cf. Table 4 of the main paper).}
    \renewcommand{\arraystretch}{0.3}
	\begin{tabular}{c|c|c|c}
		$\alpha$ & k & RMSE & ABSR \\ \hline
		0.8 & 3  & \textbf{1.80} & \textbf{0.028}  \\
		0.8 & 5  & 1.82 & 0.029  \\
		0.8 & 7  & 1.88 & 0.035  \\
		0.5 & 3  & 1.81 & 0.029  \\
		0.2 & 3  & 2.20 & 0.062   \\
		\hline
	\end{tabular}
	\label{tbl:depth-suppl}
\end{table}

\subsection{Ablation studies on $\alpha$ and $k$ in multi-modal (MM) ground truths} \autoref{tbl:depth-suppl} shows the depth estimation error on Scene Flow using \textbf{\method{CDN}}\xspace-\method{SDN}\xspace-MM, with different $\alpha$ and $k$ in preparing the MM ground truths (cf. Table 4 of the main paper). A smaller $\alpha$ leads to a larger error, which makes sense as it relies less on the ground truths.

\subsection{Learning with multi-modal (MM) ground truths}

Following subsection 4.4 and Figure 4 of the main paper, we train \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace and \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace MM for only two epochs and compare their disparity estimation performance. \autoref{fig:MM-analysis} shows the results on KITTI images. While both methods have similar predictions at smooth regions, \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace MM leads to much sharper and clearer object boundaries, suggesting that the multi-modal ground truths are better supervisions for learning around the boundaries in early epochs.

\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figures/early_image.png}
\includegraphics[width=0.6\linewidth]{figures/early.png}
\includegraphics[width=0.6\linewidth]{figures/early_mm.png}
\caption{\small \textbf{Qualitative results on learning with multi-modal ground truths at early epochs}. The top, middle, and bottom images are the left image, the result of \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace, and the result of \textbf{\method{CDN}}\xspace-\method{PSMNet}\xspace MM.}
\label{fig:MM-analysis}
\end{figure}

\subsection{Learned offsets} The offset network learns to produce the sub-grid disparity at each integral disparity values. \autoref{fig:off_visual} shows an example, in which we back-project pixels into 3D points using the estimated disparity or depth at each pixel by the \emph{mode}, with or without the offset prediction. Without the offset, the 3D points can only occupy discrete depths, leading to a discontinuous, non-smooth point cloud.

\begin{figure}
\includegraphics[width=\linewidth]{figures/offset1.pdf}
\caption{\small {A visualization of the 3D point cloud (from the bird's-eye view) derived from the estimated disparities or depths (by modes), with (right) and without (left) offset prediction.}
} \label{fig:off_visual}
\end{figure}

\subsection{Point cloud visualization}
\autoref{sfig:illustration} shows the BEV point cloud visualization. We show the 3D points generated by \method{SDN}\xspace and \textbf{\method{CDN}}\xspace-\method{SDN}\xspace as well as the ground truth LiDAR points and car/pedestrian boxes. We see that, \textbf{\method{CDN}}\xspace-\method{SDN}\xspace generates sharper points than \method{SDN}\xspace. Specifically for pixels on the foreground objects, \method{SDN}\xspace usually predicts the depths beyond the boxes due to the \emph{mean} estimates from multi-modal distributions on the boundary pixels, whereas \textbf{\method{CDN}}\xspace-\method{SDN}\xspace significantly alleviates the problem. We also see some failure cases of \textbf{\method{CDN}}\xspace-\method{SDN}\xspace: on the right image, \textbf{\method{CDN}}\xspace-\method{SDN}\xspace has a larger error on the background compared to \method{SDN}\xspace.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/pointcloud.pdf}
\caption{\small\textbf{BEV Point cloud visualization.} The blue points are obtained using \method{SDN}\xspace. The red points are from our \textbf{\method{CDN}}\xspace-\method{SDN}\xspace model. The yellow points are from the ground truth LiDAR. The green boxes are ground truth car / pedestrian locations. The observer is at the left-hand side of the point cloud looking to the right.}
\label{sfig:illustration}
\end{figure}

\subsection{Depth estimation}
Besides the Scene Flow dataset, we show the depth estimation error on KITTI Val: the 3,769 validation images for 3D object detection. We follow~\cite{you2019pseudo} to train the depth estimation model and compute the depth estimation error on pixels associated with ground truth LiDAR points. \autoref{tbl:abl_DSGN} and \autoref{sfig:converge} show the results, \textbf{\method{CDN}}\xspace-\method{SDN}\xspace achieves lower error than \method{SDN}\xspace, which explains why \textbf{\method{CDN}}\xspace-\method{SDN}\xspace (and \textbf{\method{CDN}}\xspace-\method{DSGN}\xspace) can lead to better 3D object detection accuracy. %

\begin{table}[]
	\centering
	\small
	\caption{\small \textbf{Depth error on KITTI Val.}  We compare \method{SDN}\xspace and \textbf{\method{CDN}}\xspace-\method{SDN}\xspace models.}
	\label{tbl:abl_DSGN}
	\begin{tabular}{=l|+c|+c|+c|+c}
	 & \multicolumn{2}{c}{Depth errors (m)} \\ \cline{2-5}
		  \multicolumn{1}{c|}{Method} & Mean & Median & RMSE & ABSR \\ \hline
		\method{SDN}\xspace & 0.589 & 0.128 & 3.08 & 0.044\\
		\textbf{\method{CDN}}\xspace-\method{SDN}\xspace & \textbf{0.524} & \textbf{0.093} & \textbf{3.00} & \textbf{0.042}\\
     \hline
	\end{tabular}
\end{table}

\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figures/med_err.pdf}
\caption{\small  \textbf{Depth error on KITTI Val.} We compute the median absolute depth error for different depth ranges on KITTI Val images using \method{SDN}\xspace and \textbf{\method{CDN}}\xspace-\method{SDN}\xspace.} \label{sfig:converge}
\end{figure}

\subsection{3D object detection}

We show in~\autoref{fig:AP} the object detection precision-recall curves of \method{DSGN}\xspace vs. \textbf{\method{CDN}}\xspace-\method{DSGN}\xspace. \textbf{\method{CDN}}\xspace-\method{DSGN}\xspace has higher precision (vertical) values than \method{DSGN}\xspace at different recall (horizontal) values.

\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figures/cdn_iou_dsgn.pdf}
\caption{\small We show the object detection precision-recall curves for AP$_\text{3D}$\xspace at moderate cases on cars. We compare \method{DSGN}\xspace (stereo images) and \textbf{\method{CDN}}\xspace-\method{DSGN}\xspace (stereo images).}
\label{fig:AP}
\end{figure}

\subsection{Qualitative disparity results}

We show in~\autoref{fig:depth2} and~\autoref{fig:depth3} the predicted disparity maps and the foreground errors of both \method{GANet}\xspace Deep and \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep on KITTI and Scene Flow. \textbf{\method{CDN}}\xspace generally leads to sharper and clearer object boundaries.

\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figures/pip7_1.pdf}
\caption{\small \textbf{Qualitative results on KITTI.} The top, middle, and bottom images are the left image, the result of \method{GANet}\xspace Deep, and the result of \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep, together with the foreground 3PE.}
\label{fig:depth2}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figures/pip7_3.pdf}
\caption{\small \textbf{Qualitative results on Scene Flow validation set.} The first row shows the left image and the ground truth map. The second row shows the result of \method{GANet}\xspace Deep and the result of \textbf{\method{CDN}}\xspace-\method{GANet}\xspace Deep, together with the foreground end point errors (EPE).}
\label{fig:depth3}
\end{figure}

redpajama_set_name: RedPajamaArXiv


Military in Seoul said on Monday it was closely monitoring North Korean facilities after a series of satellite images triggered international alarm that Pyongyang might be preparing a long-range missile or space launch. Analysis indicates increased activity at two key sites – the Samundong missile research facility and the Sohae rocket-testing facility, and any launch could send stuttering talks on denuclearization into disarray, AFP said. South Korea is "closely tracking and looking into all activity for possible scenarios including a missile launch" across the border, in close coordination with the US, according to Kim Joon-rak, spokesman of the Joint Chiefs of Staff.


text:
Entire Website e-Bookse-ResourcesEventsForumsJobsJournalsLinksNewsThermalpediaWho's Who
FHMT Journal
Frontiers in Heat and Mass Transfer (FHMT)
Administrator/Management
Thermal Digest
Books Guide
Journals Guide
Commercial Codes
M » Encyclopedia » Index
Boundary layer theory
From Thermal-FluidsPedia
Founding Editor (Talk | contribs)
3.Suction of the same fluid through a porous wall (for example to prevent boundary layer separation because of an adverse pressure gradient). In these cases, vw is also known and independent of temperature, but it is negative.
In general, not all mechanisms of transport phenomena (mass, momentum, and energy) occur simultaneously, even though it is possible in many practical applications. If the transport phenomena is due to momentum and heat and not mass diffusion, then eqs. (4.12) to (4.15) are uncoupled for the case of constant properties. If there is mass transfer by diffusion from the wall however, then <math>v_{w}=f\left( \omega _{1} \right)\ne 0</math>
In general, not all mechanisms of transport phenomena (mass, momentum, and energy) occur simultaneously, even though it is possible in many practical applications. If the transport phenomena is due to momentum and heat and not mass diffusion, then eqs. (4.12) to (4.15) are uncoupled for the case of constant properties. If there is mass transfer by diffusion from the wall however, then <math>v_{w}=f\left( \omega _{1} \right)\ne 0</math> and eqs. (4.12) through (4.16) are coupled, even if the properties are constant. The occurrence of mass transfer at the wall may cause vw to be nonzero and can be calculated from Fick's Law for binary mass diffusion.
The coupling effect between mass and momentum in binary mass diffusion can be easily observed by calculating mass flux at the wall as shown below:
<math>{\dot{m}}''_{1}=\rho _{1,w}v_{w}-\left. \rho D_{12}\frac{\partial \omega _{1}}{\partial y} \right|_{y=0}\,=\rho \left( \omega _{1,w}v_{w}-\left. D_{12}\frac{\partial \omega _{1}}{\partial y} \right|_{y=0} \right)</math>
where ω1,w and vw are mass fraction and the y-component of the velocity at the wall, respectively. There are five dependent variables (u, v, T, p, ω1) and two independent variables (x and y). There are five equations (4.12)-(4.16) and five unknowns with appropriate boundary conditions; therefore, it is a well posed problem.
Using scale analysis, presented in Chapter 1, order of magnitude analysis, and considering boundary layer assumptions, as noted in Section 4.3, eqs. (4.12) through (4.16) are reduced to the following forms:
Continuity equation:
<math>\frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}=0</math>
Moment equation in the x-direction:
<math>u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y}=\nu \frac{\partial ^{2}u}{\partial y^{2}}</math>
Energy equation:
<math>u\frac{\partial T}{\partial x}+v\frac{\partial T}{\partial y}=\alpha \frac{\partial ^{2}T}{\partial y^{2}}=\frac{\nu }{\Pr }\frac{\partial ^{2}T}{\partial y^{2}}</math>
Species equation:
<math>u\frac{\partial \omega _{1}}{\partial x}+v\frac{\partial \omega _{1}}{\partial y}=D_{12}\frac{\partial ^{2}\omega _{1}}{\partial y^{2}}=\frac{\nu }{Sc}\frac{\partial ^{2}\omega _{1}}{\partial y^{2}}</math>
Using boundary layer approximation for flow over a flat plate with constant free stream velocity, the momentum equation in the y-direction and the pressure gradient in the x-direction were eliminated. Eqs. (4.25) through (4.28) are now transformed into a parabolic form rather than the original elliptic form (Chapter 2). There are now four dependent variables (u, v, T, ω1) and four equations. This is much easier to solve analytically or numerically because the axial diffusion terms
<math>(\partial ^{2}u/\partial x^{2},\partial ^{2}T/\partial x^{2}\ \text{and }\partial ^{2}\omega _{1}/\partial x^{2})</math>
disappear.
== ==
1 Concepts of the Boundary Layer Theory
2 Boundary layer approximation
3 Governing Equations for Boundary Layer Approximation
Concepts of the Boundary Layer Theory
Unless the geometry is very simple, it is difficult to solve for the complete viscous fluid flow around a body. A full domain numerical solution is time consuming and impractical, because one needs to solve the full Navier-Stokes equations in the full domain, which are nonlinear, elliptic, and complex.
In 1904, Prandtl discovered that for most practical applications, the influence of viscosity is observed in a very thin domain close to the object, as shown in Fig.1. Outside this region one can assume the flow is inviscid (µ = 0) .
Figure 1: Viscous or momentum boundary layer.
The thin region where the effect of viscosity is dominant is called the momentum or viscous boundary layer. The solution of boundary layer analysis can be simplified due to the fact that its thickness is much smaller than the characteristic dimension of the object. The fluid adjacent to the surface of the body has zero relative velocity; ufluid – usurface = 0. This is also called the no slip boundary condition.
One of the assumptions of boundary layer approximations is that the fluid velocity at the wall is at rest relative to the surface. This is true except when the fluid pressure is very low, and therefore, the Kundsen number, Kn= λ/L, of the fluid molecules is much larger than 1. For external flow, the flow next to an object can be divided into two parts. The larger part is related to a free stream of fluid, in which the effect of viscosity is negligible (potential flow theory). The smaller region is a thin layer next to the surface of the body, in which the effects of molecular transport (such as viscosity, thermal conductivity and mass diffusivity) are very important.
Figure 2: Laminar and turbulent boundary layer flow over a flat plate.
Potential flow theory neglects the effect of viscosity, and therefore, significantly simplifies the Navier-Stokes equations, which provides the solution for the velocity distribution. A disadvantage of the potential theory is that since second order terms are neglected, the effect of viscosity, no slip, and impermeability boundary conditions at the surface cannot be accounted for. In general, potential flow theory predicts the free stream field accurately, despite its simplicity. Boundary layer thickness is defined as the distance within the fluid in which most of the velocity change occurs. This thickness is usually defined as the thickness in which the velocity reaches 99% of the free stream velocity, u = 0.99U∞.
Figure 2 illustrates how the momentum boundary layer thickness changes along the plate. Flow is laminar at relatively small values of x, where it is shown as the laminar boundary layer region. As x increases, the fluid motion begins to fluctuate. This is called the transition region. The boundary layer may be either laminar or turbulent in this region. Finally, above a certain value of x, the flow will always be turbulent. There is a very thin region next to the wall in the turbulent region where the flow is still laminar, called the laminar sublayer.
For flow over a flat plate, experimental data indicates
Rex ≤ 2×105 the flow is laminar
2×105 <Rex <3×106 the flow is in transition
Rex ≥ 3×106 the flow is turbulent
Boundary layer approximation
If one assumes that the boundary layer thickness, δ, is very small compared to the characteristic dimension of the object, one can make the assumption that δ is significantly less than L (δ L), where L is the characteristic dimension of the object. Using scale analysis discussed in Chapter 1, for flow over a flat plate with constant free stream velocity, one can show, in a steady two-dimensional laminar boundary layer representation, that the following conditions are met within the boundary layer region, assuming there are no body forces:
Figure 3: Boundary layer concept over a flat plate.
A similar concept exists when there is heat and/or mass transfer between a fluid and the surface of an object. Again, the region in which the effect of temperature or concentration is dominant is, in the most practical case, a region very close to the surface, as shown in Fig. 3. In general, for the case of flow over a flat plate, one can expect three different boundary layer regions with thicknesses δ, δT, and δC, corresponding to momentum, thermal, and concentration boundary layers, respectively. δ, δT, and δC are not necessarily the same thickness and their values, as will be shown later, depend on the properties of the fluid such as kinematic viscosity ( ), thermal diffusivity (α = k/ρcp), specific heat, c, and mass diffusion coefficient, D. Using boundary layer approximation, scale analysis, and order of magnitude, one can show that similar approximations exist for thermal and mass concentration boundary layer analysis.
where ω is the mass fraction.
Governing Equations for Boundary Layer Approximation
As noted before, one can obtain all pertinent information related to momentum, heat, and mass transfer between a surface and a fluid by focusing on the thin region (boundary layer) next to the surface, and solving the governing equations including the boundary conditions. This provides significant simplification irrespective of whether an analytical or numerical approach to solve the physical problem is used. For most practical applications, the effect of molecular transport due to mass, momentum, energy, and species is dominant in this thin region. The purpose of this section is to develop the transport phenomena equations for boundary layer approximation, starting from the original differential conservation equations developed in Chapter 2. Consider flow over a flat plate as shown in Figure 4.4 with constant free stream velocity, temperature, and mass fraction of U∞, T∞, and ω∞, respectively. The surface wall is kept at constant temperature and concentration. Starting from the differential conservation equations for mass, momentum, energy, and species (as presented in Chapter 2), and assuming two-dimensional, steady, laminar flow, and constant properties in a Cartesian coordinate system, the following results are obtained:
Continuity equation
Momentum equation in the x-direction
Momentum equation in the y-direction
Energy equation
Species equation for a binary system
Assuming there is a simultaneous transfer of momentum, heat and mass, the following boundary conditions are one possibility for conventional applications:
At the wall, the temperature or heat flux (or a combination of the temperature and heat flux variation) which may change along the wall, should be known. Equation (4.20) presents the two limiting cases of constant wall temperature or constant heat flux at the wall. The normal velocity at the wall is zero for the case of no mass transfer from the wall; however, there are three classes of problems in which vw ≠ 0 at the wall.
1.Mass transfer due to phase change, such as condensation or evaporation. In such cases, temperature, mass fraction, and vw are coupled at the wall through mass and energy balance at the surface of the wall.
2.Injection or blowing of the same fluid through a porous wall (for example to protect the surface from a very high temperature main stream). In these cases, vw is positive, known, and independent of temperature.
In general, not all mechanisms of transport phenomena (mass, momentum, and energy) occur simultaneously, even though it is possible in many practical applications. If the transport phenomena is due to momentum and heat and not mass diffusion, then eqs. (4.12) to (4.15) are uncoupled for the case of constant properties. If there is mass transfer by diffusion from the wall however, then and eqs. (4.12) through (4.16) are coupled, even if the properties are constant. The occurrence of mass transfer at the wall may cause vw to be nonzero and can be calculated from Fick's Law for binary mass diffusion. The coupling effect between mass and momentum in binary mass diffusion can be easily observed by calculating mass flux at the wall as shown below:
(4.24) where ω1,w and vw are mass fraction and the y-component of the velocity at the wall, respectively. There are five dependent variables (u, v, T, p, ω1) and two independent variables (x and y). There are five equations (4.12)-(4.16) and five unknowns with appropriate boundary conditions; therefore, it is a well posed problem. Using scale analysis, presented in Chapter 1, order of magnitude analysis, and considering boundary layer assumptions, as noted in Section 4.3, eqs. (4.12) through (4.16) are reduced to the following forms:
(4.25) Moment equation in the x-direction:
(4.26) Energy equation:
(4.27) Species equation:
(4.28) Using boundary layer approximation for flow over a flat plate with constant free stream velocity, the momentum equation in the y-direction and the pressure gradient in the x-direction were eliminated. Eqs. (4.25) through (4.28) are now transformed into a parabolic form rather than the original elliptic form (Chapter 2). There are now four dependent variables (u, v, T, ω1) and four equations. This is much easier to solve analytically or numerically because the axial diffusion terms
Retrieved from "http://m.thermalfluidscentral.org/encyclopedia/index.php/Boundary_layer_theory"
Editorial Portal
About Thermal-FluidsPedia
About Us | Contact Us | Terms of Use | Privacy Policy | Disclaimer
Copyright © 2010-2020 by Global Digital Central. All Rights Reserved.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Removing file extension with Rewrite in .htaccess I am currently trying to do some url rewriting. I have made some progress but I can't seem to get this to remove the file extension.
The url it outputs current looks like:
article-install-apache-on-linux.php
I would like it to look like:
article-install-apache-on-linux/
RewriteEngine on
RewriteRule ^article-(.*).php$ ./article_show_friendly.php?url=$1

A: Just remove the .php from the first part and add an optional slash /?. Also, change (.*) to (.+) to ensure 1 or more characters.
RewriteRule ^article-(.+)/?$ /article_show_friendly.php?url=$1 [L]

Also added the L flag on the end to ensure rewriting stops at that point.

redpajama_set_name: RedPajamaStackExchange


text:
package net.tenorite.protocol;

import net.tenorite.core.Tempo;
import org.junit.Test;

import static org.assertj.core.api.Assertions.assertThat;

/**
 * @author Johan Siebens
 */
public class IngameMessageTest {

    @Test
    public void raw() throws Exception {
        assertThat(IngameMessage.of().raw(Tempo.NORMAL)).isEqualTo("ingame");
        assertThat(IngameMessage.of().raw(Tempo.FAST)).isEqualTo("ingame");
    }

}

redpajama_set_name: RedPajamaGithub


text:
Taekwondo`s Karate roots
In these modern times most main stream texts regarding Taekwondo history maintains the wrong view that Taekwondo is a 2000 (sometimes its even said 5000 years) old native Korean martial art. This is probably due to the nationalist feelings in Korea after the Korean liberation from Japanese rule after world war 2. But is this the true story? I think not. At least not the whole story. Now in this post I will reveal the martial roots of the founders in the original Kwan`s.
During the japanese rule martial arts training was banned and native martial arts almost dissapeared entirely, but some Koreans who studied and worked abroad in Japan and China during the Japanese rule was allowed martial arts training. And so after the war these people came home to Korea and wanted to share their martial arts with their fellow country men, but now all things Japanese was frowned upon. The solution was to not teach where the arts came from or to make up a new history; claiming little or no influence from Japanese and Chinese martial arts and making up roots dating 2 - 5 000 years back in time.
I am not saying that in later times the Koreans has not reddiscovered their martial heritage and included it in their teachings, but in its infancy Taekwondo was really Karate as taught to the founders of the original Kwan (schools) that later formed what was to become Taekwondo. Many myths sourrounding the founders training and experience but for now I would like to give a brief summary of my own research regarding Taekwondo`s Karate roots.
It is widely known that before the name Taekwondo became the norm there were two common ways to refer to the hard style striking arts taught at the different Kwan. The two terms were "Tang Soo Do" (also written Tang Su Do etc) and Kong Soo Do (also written Kong Su Do) meaning Chinese hand and open/empty hand. What most people do not know here in the west is that these two names are not names for two different martial arts, but the Korean reading of the "Hanja" (Chinese character) used to write Karate.
Karate can be written in two ways, they have different meanings, but they are both used for the martial art now known as Karate. The first read in Korean as Tang Su Do is read as To De/To Te in Japanese and is the more traditional term for denoting the martial art of Karate. "To" in To Te can also be read as "Kara". The modern way of writing Karate is read Kara in Japanese and Kong in Korean meaning open. Open hand and Chinese hand.
The other evidence for Taekwondo Karate roots are the way they structured the training and the forms (hyung in Korean, Kata in Japanese) wich were basicly the same in both arts. Some Kwan included a few Chinese forms but for the most part they all taught essentually Shotokan Kata. This included but is not limited to: Kibon (Kicho) Hyung 1-3, Pyung ahn 1-5 Hyung, Chulgi 1-3 Hyung, and Kongsukon Hyung. The Karate Makkiwara (Dallyon Joo or Tal Yul Bong in Korean) was also in frequent use.
I will now briefly state the biggest Kwan, its founder, what style of Karate he studied, under whom and what rank he obtained in the art(s).
Chung Do Kwan Logo
Chong Do Kwan was founded in 1944 by Lee Won Kuk. He studdied Shotokan Karate under Funakoshi Gichin and Funakoshi Yoshitaka (better known as "Gigo" Funakoshi). He obtained the rank of 3rd Dan in the art, in a time when 5th Dan was the highest rank in Shotokan. Funakoshi Gichin was never given a rank higher than 5th Dan.
Song Moo Kwan logo
Song Moo Kwan was also founded in 1944 by Ro Byong Jik. He studied Shotokan Karate at the same university in Japan as Lee Won Kuk did. As such his instructors were father and son Funakoshi. Ro Byong Jik obtained 1st Dan in Shotokan Karate before returning to Korea and opening his own Dojang.
Moo Duk Kwan Logo
Moo Duk Kwan was another important Kwan. This Kwan was also founded in the 1940s. 1945 to be excact. It was founded by a man named Hwang Kee whose martial arts training is somewhat of an enigma. According to himself he studied a native Korean art called Taek Kyon which is famous in some modern Taekwondo circles as the "father art of Tae Kwon Do". This training has never ben verified and Moo Duk Kwans legacy is essentually the same as the other Kwan. Before he invented the Chil Sung forms and the Ro Youk forms there was also no sign of the other art he is said to have studied. Again according to himself he studied chinese martial arts while working in Japanese ruled China. He did in a late interview state that the source of his forms was various books on Shuri Te Karate wich explains why modern Tang Soo Do ans Soo Bahk Do include various Karate forms in their teachings as well as Hwang Kee`s own creations. Needless to say there is no formal rank that was given him in Taek Kyon, his Chinese martial art or Karate.
Chang Moo Kwan was founded in 1946 by Yon Byong In. He did not study Shotokan Karate like most other Kwan founders, but he did study a form of Karate called "Shudokan". He studied this style directly under its founder Toyama Kanken and he received the master rank of 4th Dan in this style. This makes Yon Byong In one of the highest ranked Karate masters in Taekwondo history. Some sources even states that he received 5th Dan but I can not verify it.
Only two more schools left know. The first is the Yon Moo Kwan. This school was founded in 1946 by Chun Sang Sup. I regret to say that I do not have enough material on him to state anything concrete about his martial roots, but he did found Yon Moo Kwan, made it one of the leading Kwan before dissapearing during the Korean war. In 1953 his successor reopened the school under a new name; Ji Do Kwan. This school was to become famous for its sparr ing team, and together with Moo Duk Kwan they sendt students even to Japan to compete in sparring there! According to my teacher who started his training in the Ji Do Kwan in the 1950s the Koreans won a great deal. The name of Chun Sang Sup`s successor was Yun Kwae Byung. He practised Shudokan Karate under Toyama Kanken and received a 4th Dan certificate from him. He also studied Shito Ryu Karate under thats style founder Kenwa Mabuni and from him he obtained a 7th Dan certificate. Yun Kwae Byong of the Ji Do Kwan was without a doubt the highest formal ranked practisioner of Karate in the early history of Taekwondo.
The final and last Kwan I want to include in my "brief" summary is the famous Oh Do Kwan. This Kwan is often refered to as the father Kwan of ITF Taekwondo (aka Chan Hong ryu Taekwondo). It was founded after the Korean war by General Choi Hong Hi and the Chong Do Kwan student: Nam Tae Hi. Choi Hong Hi (who dubbed himself the father of Taekwondo) is reputed to have come up with the term Taekwondo. The term Taekwondo is first documented at a meeting in 1955 when all the Kwan leaders met to discuss uniting all the Kwan to form one strong martial art for Korea. Like Hwang Kee, Choi trained Taek Kyon in his early years, but there is no way of verifying it at all. During his studies in Japan he started practising Shotokan Karate but who his instructor was is also unclear. Many sources says he studied under Funakoshi, and the sources also vary on what rank he obtained. He did at least obtain the rank of 1st Dan, but some sources say he reached the 2nd Dan level before returning to Korea.
And this completes my sumary. I hope you enjoyed it and if you have any questions regarding sources or you have something to add please feel free to add a coment:)
Lagt inn av Ørjan Nilsen
Etiketter: History
Brent Emery Pieczynski 20 July 2012 at 14:04
The claim of Shotokan being related to karate is actually a successful method of enraging Taekwondo people in America. This article will be forwarded to some man that can act innocent while everyone becomes enraged.
Ørjan Nilsen 24 July 2012 at 09:29
I do not really understand why, but yes I have enraged people before with my opinions on the relationship between Karate and Taekwondo before. I have even heard them be called unfounded and not grounded in reality.
I think this is funny because the sources of the 2000+ years old Taekwondo myth that many still believe and take as fact are not rooted in sources. They appear in textbooks that contains no sources except some old photos that show "resemblance" to Taekwondo techniques. The few sources they have on older martial arts in Korea are actually not Taekwondo allthough the term Taekwondo is often used in a generic fashion (like we might call chinese martial arts chinese boxing) to make the illusion that we are talking about the modern style named taekwondo.
The history of Taekwondo wich I have presented briefly above can be readily found in several well researched books and articles. I can name but a few: Eric Madis masterpiece (available to read at wwww.fightingarts.com), Alex Gillis "The art of Killing", Simon John O`Neill`s "Taegeuk Cipher", Kang and Lee`s "A modern history of Taekwondo" etc etc etc. What all those books/articles have in common is that they are well researched and they have the sources to back it up:-)
I do hope we do not get any "angry" or non productive comments on here though. My main sources on this article is the ones mentioned above, I hope that anyone who does not believe my article atleast can check out my sources before judging it and say it is not true:-)
Your User Name 24 February 2018 at 08:50
It is not opinion at this point, these assertions are factually backed. But, that does not invalidate Taekwondo in the least - in the truth it is still unique to a Korean origination: Karate plus a Taekkyon twist to focus on kicking.
Taekkyon was the only survivor from the Japanese occupation, no other native Korean martial art survived in form yet some survived in name only like Soobakdo.
The Japanese wasn't even the first to abolish or severely denigrate martial arts in Korean history - the Koreans themselves did a good job of doing that periodically. At times of war, martial arts thrived as a necessity, but habitually, the strong influence of the Korean interpretation of Confucianism placed great emphasis upon scholars in their caste system that looked down its nose at martial arts (often connected to Buddhism) and squelched its existence until another need arose to defend Korea again - martial arts had a yo-yo relationship with Korea for centuries in this way... And if you ask me, from my observation as a long time resident here, that yo-yo relationship is still going on right now. Mainstream TKD here is moving away from its practical combat use and plunging headlong into performance acts alongside the K-Pop phenomenon.
Ørjan Nilsen 25 February 2018 at 01:01
Thanks for taking your time to comment:-) this article was written back in 2011. If I had written it today it would have been worded differently (but the facts remains facts).
Personally I tend to be more into the practical applications to the art rather than the dancing k-pop things you see in Korea:-P The blog has nearly 300 articles to read, many of which details the history of taekwondo:-)
Great post! As a Taekwondo practitioner who has heard the myth of "ancient" Taekwondo, this only encourages me to delve further into karate and kung fu practice and research. Have you checked out Sang H. Kim's translated Muye Dobo Tongji? Very insightful link between "ancient" Korean fighting and Chinese martial arts. Anyone can see that Taekwondo (at least at its outset) looks more like Japanese karate than anything widely practiced before the occupation. Thanks for writing!
Ørjan Nilsen 12 November 2012 at 09:23
And thanks for that positive feedback:-) I do believe that some of the confusion on the "Ancient Taekwondo Myth" might have something to do with the pioneers of Taekwondo using Taekwondo as a generic term covering all Korean martial arts throughout history and as a name for our "modern" martial art at the same time. If you read older books on Taekwondo you will see that both Son Duk Sung in his Korean Karate book does this, Henry Cho in his Korean Karate Book does this and even Choi Hong Hi does it when writing about patterns.
I have read the Muye Dobo Tongji and it is a book I really like to read:-) My first Taekwondo article was about that very book (I posted it on my blog in two parts: http://jungdokwan-taekwondo.blogspot.no/2011/10/muyedobotongji-oldest-kma-manual.html ).
The likeness with Japanese Karate was what made me check out the "Karate link" in the first place. We do have more kicks and depending on the organisation/Club/Instructor higher and more flashy kicks than main stream Karate but our forms, basic techniques, 1, 2, and 3 step sparring all seemed to correspond to Karate.
If you liked this article you might want to check out the series on the original Hyung (forms) that the different Kwan used in the early 60s before they made their own inventions. The first one in the series can be read here: http://jungdokwan-taekwondo.blogspot.no/2011/11/part-one-dan-promotion-test-in-1962.html .
British Communication Academy 28 August 2017 at 17:06
I found your this post while searching for some related information on blog search...Its a good post..keep posting and update the information.
Wonderful article, thanks for putting this together! This is obviously one great post. Thanks for the valuable information and insights you have so provided here. aikido
Alex Ellis 28 April 2018 at 04:32
Ørjan Nilsen 29 April 2018 at 18:55
Thank you Alex:-)
N.D 3 May 2020 at 11:40
One thing about Taekkyeon, which people often credit for TKD's kicking...
Having studied Taekkyeon a bit in Korea, I can tell you Taekkyeon is much, much closer to Chinese Shuai Jiao in spirit.
Meaning, it places a heavy emphasis on flooring the opponent. Kicks are almost incidental in this. It is really more of an art of sweeping and tripping.
It is not surprising.
In that sense it shares a lot with other arts used for self defense: avoiding touching the "dirty" attacker with your hands, and focusing on flooring an opponent to get the upper hand or fleeing.
The French art of Savate also heavily emphasizes low level kicks and sweeps for that reason.
A floored aggressor can't run after you immediately, may get hurt falling down, and will be in a position of weakness to be struck or kicked.
TKD's kicks come from Gigo Funakoshi's Shotokan.
They used to be identical. They only changed because of competition, to emphasize speed at the expense of all other attributes.
Taekkyeon kicks are done with the heel, the ball of the foot. But mostly the heel - it was obviously designed for kicking with soft shoes / sandals on.
The instep kick is seldom used, mostly as a crescent kick or kick to the groin.
Recommended by me (Amazon Affiliated link)
Practical application of Poomsae
Poomsae
Kyorugi
Short discussion on the learning of Poomsae
I have thought about this on and off for a very long time now. Actually I started thinking about this at the very beginning of when it dawne...
What was old hard style TKD like?
I often get the question of how old hard style Taekwondo was like. This is a good question and I must say it is very difficult to have an ac...
Geom or Do? How language changes in martial arts
Hi there :-) I have been going full Muyedobotongji rabbit hole these last few months :-P I have posted a lot of stuff on facebook, and one o...
The importance of "Chambering"
 Traditional Martial Artists often discuss "technique" and how to properly excecute them. Great deal of thought and teachin...
Palgwe forms (+ original Koryo as a bonus)
I often see questions about the Palgwe forms on internet forums. The most common ones are how were they done/ how did they look like? Are th...
Another long lost form: Chulgi hyung
Please read this: Stop with the basterised bowing!
The hand on the hip? Why?
Where is the interest in Poomsae applications?
Label disease
A closer look at Poomsae training.
A closer look at Poomsae training part two
Taekwondo Time Travel.
Funny Taekwondo videos
"Teacher I know all the Poomsae in our system! Wha...
Wim Demeeres Blog
Podcast Episode 111: Ares- Lineage
Soo Shim Kwan 水心館수심관
Beating Swords into Ploughshares: Pondering Peace and Martial Arts
THE SINGLE MOUNTAIN BLOCK AND ITS MISUNDERSTANDINGS
The Way of Least Resistance
The importance of basics
Traditional Taekwondo Techniques Blog
Training So Hard Angels Would Cry
Foot Fist Way
49. Cheonkwon slapping blocks
Taekwondo Moo Duk Kwan
Taekwondo for Self Defense?
Tactical Taekwondo
Martial words: Master
KARATEbyJesse
Learn Traditional Kobudo Weapons From Okinawa
The Mortal Mouse
I was a shattered person but martial arts is helping me re-build again!
Bunkai Jutsu
White Dragon Taekwondo Blog
Why I Will Always Refuse To Teach Martial Arts For Free And Why You Shouldn't Either
Ryukyu Martial Arts
Motobu-Inspired Hand Trapping
Teach Combatives
Motor Learning for Martial Arts [Intro]
Taekwondo Fighting Heaven and Earth
All Warfare is Based on Deception
Tuite Jutsu (Personal training/teaching methodologies in Te)