text:
\section{Experiments}\label{sec:experiments}
We test the validity of our proposed 3D confidence measures by considering the deep architectures implemented in two common PL\xspace methods, \ie the first PL\xspace method, \textit{Wang et al.}~\cite{wang2019pseudo}, and a current top-performing state-of-the-art method, PatchNet~\cite{Ma_2020_rethink}. We modify their architectures to include our proposed \textit{3D Confidence Head}, which in all our experiments is implemented mirroring the existing \textit{3D Head}. In detail, it is implemented as one set of fully-connected layers for \cite{wang2019pseudo} and three distance-specific fully-connected modules for \cite{Ma_2020_rethink}. We follow the schedules and hyperparameters choices of \cite{Ma_2020_rethink} and ~\cite{wang2019pseudo}, with the only addition of the 3D Confidence loss which is given a weight of 1.0. Additional implementation details are given in Sec.~\ref{sec:impl_details}.
We follow the experimental protocol of all PL\xspace-based works and evaluate our method on the KITTI3D~\cite{Geiger2012CVPR} benchmark.
All the results presented and reported in this work have been computed with the official $AP|_{R_{40}}$ metric.

\begin{table}[tb!]
    \centering
    \resizebox{!}{0.55\columnwidth}{
    \begin{tabular}{l|ccc}

        \toprule

        & \multicolumn{3}{c}{\textit{Car 3D AP}} \\

        Method & Easy & Mod. & Hard \\

        \midrule

        OFTNet~\cite{Roddick18}            	    &  1.61 &  1.32 &  1.00 \\
        FQNet~\cite{Liu+19}                     &  2.77 &  1.51 &  1.01 \\
        ROI-10D~\cite{Manhardt_2019_CVPR} 	    &  4.32 &  2.02 &  1.46 \\
        GS3D~\cite{li2019gs3d}                  &  4.47 &  2.90 &  2.47 \\
        MonoGRNet~\cite{qin2019monogrnet}  	    &  9.61 &  5.74 &  4.25 \\
        Wang et al.~\cite{wang2019pseudo}       &  9.87 &  6.40 &  5.46 \\
        MonoDIS~\cite{simo2019monodis}          & 10.37 &  7.94 &  6.40 \\
        MonoPSR~\cite{ku2019monopsr}            & 10.76 &  7.25 &  5.85 \\
        Mono3D-PL~\cite{weng2019mono3dppl}      & 10.76 &  7.50 &  6.10 \\
        SS3D~\cite{jorgensen2019ss3d}           & 10.78 &  7.68 &  6.51 \\
        MonoPair~\cite{li2020monopair}          & 13.04 &  9.99 &  8.65 \\
        SMOKE~\cite{liu2020smoke}               & 14.03 &  9.76 &  7.84 \\
        RTDM3D$^\textbf{A}$ ~\cite{li2020rtm3d} & 14.41 & 10.34 &  8.77 \\
        M3D-RPN~\cite{brazil2019m3d}       	    & 14.76 &  9.71 &  7.42 \\
        MoVi-3D~\cite{simo2020virtual}          & 15.19 & 10.90 &  9.26 \\
        PatchNet~\cite{Ma_2020_rethink}         & 15.68 & 11.12 & 10.17 \\
        AM3D~\cite{ma2019am3d}                  & 16.50 & 10.74 & 9.52 \\
        MonoDIS~\cite{simo2020pami}             & 16.50 & 12.20 & \underline{10.30} \\
        D4LCN~\cite{ding2020d4lcn}              & 16.65 & 11.72 & 9.51 \\
        Liu et al.$^\textbf{A}$ ~\cite{yuxuan2021ral} & \underline{21.65} & \textbf{13.25} & 9.91 \\

        \midrule

        Our PatchNet & \textbf{22.40} & \underline{12.53} & \textbf{10.64} \\

        \bottomrule
    \end{tabular}
    }
    \caption{Test set SOTA $AP|_{R_{40}}$ official results on KITTI3D for class \textit{Car}. Best scores in bold, runner-ups underlined. \textbf{A}~=~trained with additional data and only on class \textit{Car}.}
    \label{tab:kitti_test_car}
\end{table}

\begin{table}[tb!]
    \centering
    \resizebox{0.99\columnwidth}{!}{
    \begin{tabular}{l|ccc|ccc}

        \toprule

        & \multicolumn{3}{c|}{\textit{Cyclist 3D AP}} & \multicolumn{3}{c}{\textit{Pedestrian 3D AP}} \\

        Method & Easy & Mod. & Hard & Easy & Mod. & Hard \\

        \midrule

        M3D-RPN~\cite{brazil2019m3d}       	    & 0.94 & 0.65 & 0.47 & 4.92 & 3.48 & 2.94 \\
        MoVi-3D~\cite{simo2020virtual}          & 1.08 & 0.63 & 0.70 & \underline{8.99} & \underline{5.44} & \underline{4.57}\\
        MonoDIS~\cite{simo2020pami}             & 1.17 & 0.54 & 0.48 & 7.79 & 5.14 & 4.42 \\
        D4LCN~\cite{ding2020d4lcn}              & 2.45 & 1.67 & 1.36 & 4.55 & 3.42 & 2.83 \\
        SS3D~\cite{jorgensen2019ss3d}           & 2.80 & 1.45 & 1.35 & 2.31 & 1.78 & 1.48 \\
        MonoPair~\cite{li2020monopair}          & \underline{3.79} & \underline{2.12} & \underline{1.83} & \textbf{10.02} & \textbf{6.68} & \textbf{5.53} \\

        \midrule

        Our - PatchNet & \textbf{7.79} & \textbf{4.32} & \textbf{3.98} & 3.00 & 1.81 & 1.59 \\

        \bottomrule
    \end{tabular}
    }
    \caption{Test set SOTA $AP|_{R_{40}}$ official results on KITTI3D of published multi-class methods for \textit{Cyclist} and \textit{Pedestrian}. Best scores in bold, runner-ups underlined.}
    \label{tab:kitti_test_cycped}
\end{table}

\myparagraph{Experimental results.}
In Tab.~\ref{tab:kitti_ablations} we investigate the influence of the {3D confidence} on the PL\xspace-LiDAR methods \cite{wang2019pseudo} and~\cite{Ma_2020_rethink}. In particular, we compute the 3D object detection metrics on the validation and test set of KITTI3D with the baseline methods as well as with the addition of the \textit{3D Confidence Head} (\textit{+ 3D Confidence}) trained with both the {absolute} and {relative} learning procedure. As shown in the table, we observe a major improvement on the $3D$ $AP$. This
validates our hypothesis about the importance of having a 3D confidence prediction component in PL\xspace-based methods. The {relative 3D confidence} also consistently outperforms the {absolute} one, demonstrating the validity of our proposed relative formulation.
In Tab.~\ref{tab:kitti_test_car}, \ref{tab:kitti_test_cycped} we compare our results with state-of-the-art methods on the KITTI3D test set.
Our method based on PatchNet structure achieves state-of-the-art performance on the classes \textit{Car} and \textit{Cyclist}, while it does not surpasses previous approaches on the  \textit{Pedestrian}. We ascribe this behaviour to the fact that instances of the class \textit{Pedestrian} are extremely rare in the \textit{Eigen} depth training set, thus having a negative impact on the depth map quality.

\begin{table}[t]
    \centering
    \resizebox{0.99\columnwidth}{!}{
    \begin{tabular}{l|ccc|ccc}
        \toprule
          & \multicolumn{3}{c|}{\textit{Validation} $3D~AP$} & \multicolumn{3}{c}{\textit{Test} $3D~AP$}  \\

        Method & Easy & Mod. & Hard & Easy & Mod. & Hard \\
        \midrule

        PatchNet                     & 20.79 & 10.55 &  8.90 & 10.88 &  7.42 & 6.51 \\
        \textit{+ Abs. 3D Conf.}     & 23.37 & 15.49 & 12.70 & 17.38 & 10.30 & 8.78\\
        \textit{+ Rel. 3D Conf.}     & \textbf{24.51} & \textbf{17.03} & \textbf{13.25} & \textbf{17.69} & \textbf{10.85} & \textbf{9.37} \\

        \bottomrule
    \end{tabular}}
    \caption{Validation and Test set $AP|_{R_{40}}$ results on KITTI3D obtained with the BTS~\cite{lee2019bts} depth estimator trained on our \textit{GeoSep} depth training split.}
    \label{tab:kitti_ablations_bias}
\end{table}

\begin{table}[t]
    \centering
    \resizebox{0.80\columnwidth}{!}{
    \begin{tabular}{c|ccc|ccc}

        \toprule

        & \multicolumn{3}{c|}{ \textit{Wang et al. Car} 3D AP} & \multicolumn{3}{c}{\textit{PatchNet Car} 3D AP} \\

        $\beta$ & Easy & Mod. & Hard & Easy & Mod. & Hard \\

        \midrule

        0.1 & 28.51 & 18.78 & 15.85 & 33.69 & 21.00 & 16.42\\
        1 & 32.44 & 20.84 & 17.26 & 37.04 & 23.26 & 18.78\\
        10 & 30.72 & 19.82 & 16.43 & 32.05 & 19.06 & 15.47\\

        \bottomrule

    \end{tabular}}
    \caption{\textit{Absolute 3D Confidence} ablation results.}
    \label{tab:kitti_ablations_temp}
\end{table}

In Tab.~\ref{tab:kitti_ablations_bias} we also report the results of \textit{PatchNet + 3D Confidence} obtained by relying on a depth estimator trained on the \textit{GeoSep} depth training set. Again, this table demonstrates the merit of our contributions in term of 3D confidence estimation. Additionally, relatively to our first contribution, we notice that a gap between the validation and test set results still exists, indicating the bias issue. However, this does not invalidate our observations regarding the effectiveness of the proposed 3D confidence.

In Tab.~\ref{tab:kitti_ablations_temp} we perform a sentitivity study and analyze the behaviour of the absolute 3D confidence with respect to changes in the temperature value $\beta$. We chose a temperature of 0.1, 1 and 10 and computed results on the KITTI3D validation set. The significant change in performance demonstrates that this type of {absolute} confidence is indeed sensitive to hyperparameter tuning.

\section{Implementation details}\label{sec:impl_details}
In this section we provide details about the implementation and additional information about the hyper-parameters. Since our method is subdivided into multiple branches, we provide details of each one namely \textit{2D Detection}, \textit{Pseudo-LiDAR} and \textit{3D Detection}. In all our experiments, we trained our models on a single NVIDIA GTX 1080 Ti with 11GB of memory.

\myparagraph{2D Detection.}
As described in Sec.~\ref{sec:experiments} we do not train a 2D detector but instead rely on pre-computed 2D detections. In our experiments we used, for both validation and test set, the 2D detections used in PatchNet~\cite{Ma_2020_rethink}.

\myparagraph{Pseudo-LiDAR.}
We took the open-source code of BTS~\cite{lee2019bts} and selected the DenseNet161-based estimator. For our results on the Eigen \etal~\cite{Eigen2015} we used the model trained by the authors\footnote{\url{https://cogaplex-bts.s3.ap-northeast-2.amazonaws.com/bts_eigen_v2_pytorch_densenet161.zip}}. For the training on our novel \textit{GeoSep} splits, we used the ImageNet~\cite{Deng2009} pre-trained model and followed the official schedule and hyperparameters.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\columnwidth]{pipeline_3d.pdf}
    \caption{Example of final part of the 3d detection architecture, where we introduced our proposed \textit{3D Confidence Head} in PatchNet~\cite{Ma_2020_rethink}. The implementation of the \textit{3D Confidence Head} (red) follows the one of the \textit{3D Head} (blue).}
    \label{fig:3d_conf_head}
    \vspace{-12pt}
\end{figure}

\myparagraph{3D Detection.}
The architecture of our proposed models, \ie the ones based on Wang \textit{et al.}~\cite{wang2019pseudo} and PatchNet~\cite{Ma_2020_rethink}, always follow the official one with the only exception of the introduction of our proposed \textit{3D Confidence Head}. The implementation of this particular head closely follows the one of the respective 3D Head. In particular, for our implementation based on Wang \textit{et al.}~\cite{wang2019pseudo} we introduced a series of three fully-connected layers with \textit{512-D}, \textit{512-D}, and \textit{1-D} dimensions respectively. For the implementation of PatchNet we introduced three distance-specific heads composed by a series of three fully-connected layers with \textit{512-D}, \textit{512-D}, and \textit{1-D} dimensions respectively. We depict the PatchNet \textit{3D Head} (blue), along with our proposed \textit{3D Confidence Head} (red), in Fig.~\ref{fig:3d_conf_head}. We trained our model with the Adam optimizer with a learning rate of 0.001 and a batch size of 64 for 100 epochs, decreasing the learning rate by a factor of 0.1 at the $20^{th}$ and $40^{th}$ epoch.

\section{Additional Qualitative results}
After a initial description of the visualization method, we provide additional qualitative results of our detections on KITTI3D.

\myparagraph{Visualization method.}
We visualize our results by super-imposing our \textit{PatchNet + Relative 3D Confidence} 3d bounding box detections on the input RGB image as well as rendered Pseudo-LiDAR pointcloud, as presented in Fig.~\ref{fig:video}.

\begin{figure}[h!]
    \centering
    \resizebox{0.99\columnwidth}{!}{
        \includegraphics[width=\textwidth]{video.png}
    }
    \caption{Example of output visualization. Top: Visualization of our predictions on the colored Pseudo-LiDAR pointcloud. Bottom: Visualization of our predictions, with corresponding confidence score, on the input RGB image.}
\label{fig:video}
\end{figure}

In the top part of the images we visualize our detections on the rendered Pseudo-LiDAR pointcloud, where each point has been colored with its corresponding RGB value (if available), and consequently visualized our predicted 3d bounding-boxes in green for \textit{Car}, cyan for \textit{Cyclist} and red for \textit{Pedestrian}. The presence of black pixels (\eg on top of objects) is due to the fact that we rendered the scene from a point-of-view which is different from the one of the KITTI3D RGB camera. This change of pose inevitably introduces these black pixels on regions which were not visible from the RGB camera pose.

\myparagraph{Qualitative results on images.}
In Fig.~\ref{fig:ablation},\ref{fig:ablation2} we show our results on KITTI3D test set images. Our proposed confidence is demonstrated to reliably determine the overall quality of the predicted 3D bounding box. The confidence is in fact higher on nearer and not occluded objects, \ie where the estimation is more reliable, and seems to degrade with distance and occlusions. We also included some failure cases in which our confidence is shown to be less reliable. In particular, we have identified some imprecise or empty detections that still have fairly high confidence.

\myparagraph{Qualitative results on video sequences.}
We further provide a qualitative video\footnote{\url{https://youtu.be/Sp8tvP6KX44}} by showing our predictions on complete KITTI3D sequences taken from the KITTI3D validation set. Unfortunately, it was not possible to provide videos on the KITTI3D test set sequences due to the unavailability of test set sequence information. The predictions are computed for each frame in an independent manner, without exploiting temporal information in any way. Despite the presence of failure cases, \eg when object are too near/far/occluded, our confidence score is shown to generally reflect the quality of the 3d detection.

\begin{figure*}[h]
    \centering
    \vspace{30pt}
    \includegraphics[width=2.0\columnwidth]{000026.png}
    \includegraphics[width=2.0\columnwidth]{000178.png}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation}
    \vspace{30pt}
\end{figure*}
\begin{figure*}[h]
    \centering
    \vspace{30pt}
    \includegraphics[width=2.0\columnwidth]{000175.png}
    \\ ~ \\
    \includegraphics[width=2.0\columnwidth]{002784.png}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation2}
    \vspace{30pt}
\end{figure*}

\section{Preliminaries}
We first review the monocular 3D object detection task and introduce the KITTI dataset~\cite{Geiger2012CVPR} -- the most influential benchmark to assess the performance of 3D detection methods. We also report the results of an experimental analysis on KITTI, highlighting the crucial role of depth estimation on the performance of state-of-the-art PL\xspace-based methods.

\subsection{Monocular 3D Object Detection}\label{sec:3d_detection}
The monocular 3D object detection task consists in detecting and localizing all the visible objects of interest (\eg cars) by means of 3D bounding boxes given a single RGB image as input.
Localization must be done in 3D space, properly estimating the 3D coordinates (in meters) of the center of the object $O_i=(X_i, Y_i, Z_i)$, where $X_i, Y_i$ are related to the horizontal and vertical translations, respectively, and $Z_i$ is the distance of the object's center from the camera. The localization also includes the estimation of the object's metric shape $S_i=(H_i, W_i, L_i)$ representing the object's height, width and length, as well as the object's rotation $R_i$ \wrt the camera reference system. The detection requires also to estimate a confidence value $C_i$ which generally reflects the quality and determines how confident the detector is about the particular 3D detection. In this monocular setting, it is common to assume to have a calibrated camera and to know the corresponding intrinsic camera parameters.

\subsection{The KITTI Dataset}\label{sec:kitti}
The KITTI Dataset comprises a broad collection of data from street-level sequences, captured with a multi-sensor setup in the city of Karlsruhe (Germany) in 2011. The remarkable diversity of the sensors enabled many benchmarks, including \textit{3D object detection} and \textit{depth estimation}, which are most relevant for this work.

\myparagraph{KITTI 3D object detection benchmark (KITTI3D).}
To our knowledge, all 3D object detection methods, and in particular monocular image-based ones, adopted KITTI3D as their predominant, and usually exclusive, testing field. The KITTI3D benchmark is composed of an official \textit{training} and \textit{testing} split, comprising 7481 and 7518 images, respectively. Following \textit{Chen \etal}~\cite{chen2015validationdetection}, it is common to split the training set into \textit{unofficial} training and validation splits, with 3712 and 3769 images, respectively. KITTI provides 2D and 3D bounding box annotations for \textit{Cars}, \textit{Pedestrians} and \textit{Cyclists}, and each box is assigned to one of the \textit{difficulty} levels \textit{Easy}, \textit{Moderate} or \textit{Hard}, depending on the object's 2D height ($\approx$~object's distance), degree of occlusion, and truncation.
KITTI3D adopts two main evaluation metrics, \ie, \textit{3D Average Precision (3D AP)} and \textit{Bird's Eye View Average Precision (BEV AP)}. As reported in~\cite{simo2020pami}, AP$|_{R_{40}}$ is the only legitimate 3D detection AP score, deprecating the previously used AP$|_{R_{11}}$ score.

\myparagraph{Depth prediction benchmark.}
The KITTI depth prediction benchmark offers \textit{official} training and testing splits, but it is common to split~\cite{Eigen2015} the training data into \textit{unofficial} training and validation sets of 23488 and 697 images, respectively. Depth prediction methods are inferring pixel-specific distance estimates \wrt the camera and are evaluated with several metrics like \textit{Absolute Relative Error (AbsRel)}, \textit{Squared Relative Error (SqRel)}, \etc.

\subsection{The Crucial Role of Depth}\label{sec:oracle}
We also provide the results of an \textit{oracle} analysis demonstrating that depth is the most influential factor for performance in monocular 3D object detection.
Following the definitions in Sec.~\ref{sec:3d_detection}, we used KITTI3D predictions of state-of-the-art monocular 3D object detection methods~\cite{brazil2019m3d, Ma_2020_rethink, simo2019monodis,wang2019pseudo} and compared their 3D object detection performances by \textit{substituting} sub-task predictions (\eg \textit{depth}) with their corresponding ground-truth values.
In Tab.~\ref{tab:biased_oracle} we show that certain sub-tasks like \textit{rotation} ($R$) and \textit{shape} ($W,H,L$) prediction, despite the substitution with ground-truth values, do not significantly improve performance. In contrast, substituting the predicted \textit{depth} estimation ($Z$) with ground truth improves substantially, meaning that \textit{depth} is by-far the most crucial component for 3D object detection. Notably, this observation is consistent for all the different tested methods.

\begin{table}[t]
    \centering
    {\footnotesize
    \resizebox{0.99\columnwidth}{!}{%
    \begin{tabular}{l|c|ccc|ccc}
        \toprule

         & Oracle & \multicolumn{3}{c|}{M3D-RPN~\cite{brazil2019m3d}} & \multicolumn{3}{c}{MonoDIS~\cite{simo2020pami}} \\

        Category & sub-task & Easy & Mod. & Hard & Easy & Mod. & Hard\\

        \midrule

        \multirow{5}{*}{RGB-based} & -- & 12.78 & 10.36 & 8.07 & 16.71 & 12.32 & 10.58 \\
         & $\hat{R}$ & 14.71 & 11.78 & 9.26 & 17.27 & 12.76 & 11.45 \\
         & $\hat{HWL}$ & 13.47 & 10.52 & 8.26 & 16.75 & 12.56 & 11.29 \\
         & $\hat{XY}$ & 22.63 & 17.47 & 13.48 & 29.59 & 22.17 & 19.31 \\
         & $\hat{Z}$ & \textbf{34.53} & \textbf{28.35} & \textbf{22.51} & \textbf{45.99} & \textbf{38.02} & \textbf{33.48}\\

        \midrule
        \midrule

         & Oracle & \multicolumn{3}{c|}{\textit{Wang et al.}~\cite{wang2019pseudo}} & \multicolumn{3}{c}{PatchNet~\cite{Ma_2020_rethink}} \\

        Category & sub-task & Easy & Mod. & Hard & Easy & Mod. & Hard\\

        \midrule

        \multirow{5}{*}{Pseudo-LiDAR-based} & -- & 23.71 & 12.40 & 10.61 & 31.15 & 16.23 & 13.49\\
         & $\hat{R}$ & 24.04 & 13.39 & 11.13 & 31.60 & 17.43 & 14.58\\
         & $\hat{HWL}$ & 25.73 & 14.50 & 11.64 & 34.19 & 19.01 & 15.58 \\
         & $\hat{XY}$ & 33.76 & 20.37 & 17.22 & 44.23 & 25.62 & 21.76\\
         & $\hat{Z}$ & \textbf{53.71} & \textbf{35.15} & \textbf{29.38} & \textbf{59.81} & \textbf{41.93} & \textbf{35.94}\\

        \bottomrule

    \end{tabular}}}
    \caption
    \textbf{Oracle analyses}. We computed the object detection results (\textit{Car} 3D $AP|_{R_{40}}$) of state-of-the-art methods by substituting selected predicted components (\textit{Oracle}) with their corresponding ground-truth value (\eg $\hat Z$).
    }
    \label{tab:biased_oracle}
\end{table}

\section{The Bias in Pseudo-LiDAR Experiments}
With depth identified as most critical component in monocular 3D detection works, it becomes obvious that PL\xspace-based methods are particularly sensitive to inputs from depth estimators trained in a biased way.

\begin{table}[t]
    \centering
    \resizebox{1.0\columnwidth}{!}{
    \begin{tabular}{l|c|ccc|ccc}
        \toprule
        3D Object & Depth & \multicolumn{3}{c|}{\textit{Validation set}~~$3D~AP\uparrow$} & \multicolumn{3}{c}{\textit{Test set~~$3D~AP\uparrow$}}  \\

         Detector & Estimator & Easy & Mod. & Hard & Easy & Mod. & Hard \\
        \midrule
        \midrule

        \textit{Wang et al.} ~\cite{wang2019pseudo} & BTS Eigen    & 24.47 & 13.40 & 10.92 & 9.87 & 6.40 & 5.46 \\
        PatchNet~\cite{Ma_2020_rethink} & BTS Eigen                & 31.60 & 18.22 & 15.10 & 14.00 & 8.70 & 7.39  \\

        \midrule

        \textit{Wang et al.} ~\cite{wang2019pseudo} & BTS GeoSep  & 17.20 &  9.35 &  7.57 & 10.76 & 6.86 & 5.93 \\
        PatchNet~\cite{Ma_2020_rethink} & BTS GeoSep              & 20.79 & 10.55 &  8.90 & 10.88 & 7.42 & 6.51 \\

        \bottomrule
    \end{tabular}}
    \caption{Pseudo-LiDAR results on KITTI3D validation and official benchmark, class \textit{Car}, official $\text{AP}|_{R_{40}}$ metric.}
    \label{tab:detection_bias}
    \vspace{-10pt}
\end{table}

\begin{table}[t]
    \centering
    \resizebox{1.0\columnwidth}{!}{
    \begin{tabular}{l|l|c|c|c|c}
        \toprule
        Train set & Validation Set & $d_1\uparrow$ & $AbsRel\downarrow$ & $RMSE\downarrow$ & $SILog\downarrow$ \\
        \midrule

        Eigen & Eigen validation     & 0.908 & 0.084 & 4.003 & 16.577 \\
        Eigen & Detection training   & 0.926 & 0.067 & 3.806 & 15.250 \\
        Eigen & Detection validation & 0.920 & 0.072 & 3.838 & 16.063 \\

        \midrule

        GeoSep & GeoSep validation    & 0.904 & 0.093 & 3.627 & 14.019 \\
        GeoSep & Detection training   & 0.858 & 0.111 & 4.830 & 15.960 \\
        GeoSep & Detection validation & 0.872 & 0.105 & 4.429 & 15.872 \\

        \bottomrule

    \end{tabular}}
    \caption{Depth estimation results with BTS on KITTI, computed \wrt ground-truth depth obtained from LiDAR scans.}
    \label{tab:depth_bias}
    \vspace{-10pt}
\end{table}

\subsection{The Source of the Bias}
To our knowledge, all PL\xspace-based methods published so far were exclusively evaluated on the KITTI3D~\cite{Geiger2012CVPR} dataset which, as described in Sec.~\ref{sec:kitti}, shares data among several benchmarks like 3D object detection and depth prediction. With the advent of PL\xspace, it is however paramount to identify potential sources of cross-pollination in task-specific dataset splits. Our investigations showed that previous, PL\xspace-based works
~\cite{wang2019pseudo, yang2019pseudopp, Ma_2020_rethink} were built on top of DORN~\cite{FuCVPR18-DORN}, \ie a state-of-the-art depth estimator, that in turn however included a majority of images from the detection \textit{validation} set during its training.
Specifically, we found \textbf{1226/3769} (\textbf{32.5\%}) images to be shared between the widely adopted Eigen~\etal training split~\cite{Eigen2015} for depth estimation and the
commonly used Chen \etal~\cite{chen2015validationdetection} validation split for 3D object detection. When adding also the images belonging to the same capturing sequence, the numbers slightly increase to 1258/3769 (33.4\%).

We illustrate the full extent of the contamination in Fig.~\ref{fig:map}, plotting GPS positions and hence the overlap of the different splits (Eigen \etal depth training split in black; Chen \etal validation split for 3D object detection in red).

In Tab.~\ref{tab:detection_bias}, we show the effect of the contamination on the validation and test scores for two state of the art PL\xspace methods ~\cite{wang2019pseudo,Ma_2020_rethink}. The rows corresponding to \emph{Eigen} are based on biased depth as input, which was generated using BTS~\cite{lee2019bts} trained on the Eigen \etal split. We rely on BTS~\cite{lee2019bts} because it represents a novel state-of-the-art depth estimator\footnote{Differently from the depth estimator usually used by PL\xspace methods, \ie DORN~\cite{FuCVPR18-DORN}, for which official training code is not publicly available, BTS provides a complete open-source code (\url{https://github.com/cogaplex-bts/bts})}. The huge performance drops (up to 17.6 AP) between obtained validation and test scores clearly indicate the relevance of the bias issue discussed here.

\subsection{Can the Bias be Removed?}\label{sec:unbiased_pl}
As stated above, a contamination exists between the depth training and detection validation sets used by PL\xspace-based methods. To further support our hypothesis that this contamination causes a bias in the KITTI3D validation scores, we introduce a geographical separation between the two data sets. We create a novel depth training set, \textit{GeoSep}, enforcing significant spatial separation between the datasets used for depth estimation and detection.
Exploiting the GPS information included in the available KITTI benchmark data, we create two novel train/val depth splits by withholding all images i) captured closer than 200m from any KITTI3D training or validation detection image, and ii) belonging to any of the KITTI3D detection sequences.
From a total available amount of 47962 images the aforementioned filtering process yields 22954 images, which we divide in 22287 for the training set and 667 images for the validation set. The distance threshold has been chosen to ensure that our novel \textit{GeoSep} splits would have approximately the same number of images of the Eigen \etal~\cite{Eigen2015} splits, which are 23488 for training and 697 for validation, respectively.
Our new \textit{GeoSep} data split is visualized in Fig.~\ref{fig:map} (green markers), showing a clear safety margin between depth training (Eigen \etal~\cite{Eigen2015}, black markers) and object detection validation (red markers) splits.

\begin{figure}[t]
    \centering
    \resizebox{0.99\columnwidth}{!}{
    \includegraphics[width=\columnwidth]{new_map.pdf}}
    \caption{Geographical distribution of the biased training (black), detection validation (red) and geographically separated (green) depth training splits. Square boxes highlight parts where the overlap between the biased depth training and detection validation sets are particularly evident.}
    \label{fig:map}
    \vspace{-12pt}
\end{figure}

To verify whether our novel split solves the bias issue, we use it to train a depth estimation model. We use BTS~\cite{lee2019bts} as depth prediction network and, again, consider the state-of-the-art PL\xspace methods in~\cite{wang2019pseudo} and~\cite{Ma_2020_rethink}. The results of our analysis, shown in Tab.~\ref{tab:depth_bias} and Tab.~\ref{tab:detection_bias}, still indicate the presence of a bias in both depth estimation and 3D detection results.
To our great surprise, despite the lack of geographical intersection between the training splits, the gap between validation and test results is still substantially higher (up to 10 AP) compared to the gap that methods using RGB-only inputs typically incur ($\approx$ 3-5 AP).
This suggests a more structured form of contamination that goes beyond the simple geographical distribution of the data, perhaps related to intrinsic factors such as the visual appearance and semantic similarity of the scenes (\eg presence of similar streets).

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.99\columnwidth]{detections.png}
    \caption{Qualitative results of our method with confidence scores of each detection. Top: We report the 2D confidence score that PL-based methods typically use. Bottom: We report the learned 3D confidence predicted by our method.}
\label{fig:detections}
\end{figure}

The persistence of the bias using both depth training splits (Eigen or \textit{GeoSep}) make us draw the conclusion that fair comparisons should not, at least in these settings, be performed on the KITTI3D validation set.
On the other hand, the fact that published PL\xspace-based methods are not able to surpass state-of-the-art RGB-only based methods (see Tab.~\ref{tab:kitti_test_car} first block) is an indication that the test set itself does not suffer from the same type of bias, thus preserving its validity for the sake of fair comparisons.
Following these conclusions, all the comparisons related to our second contribution will be made on the official test set while the validation set will be used for ablation studies.
Despite our study only partially identifies the source of the bias, this work provides the first analysis of the issue revealing potentially unfair comparisons and we encourage the community to take it into account for future works.

\begin{figure*}[t]
    \centering
    \includegraphics[width=1.99\columnwidth]{pipeline.pdf}
    \caption{Architecture of a generic Pseudo-LiDAR-based method integrating the proposed 3D confidence component.}
    \label{fig:pipeline_all}
    \vspace{-12pt}
\end{figure*}

\section{Introduction}\label{sec:intro}
By providing information about pose, location and category of objects in the 3D space,
3D object detection constitutes an enabling technology for applications like autonomous driving or augmented reality.
To obtain accurate localisation performance, existing solutions rely on depth information inferred from stereo cameras or derived from Light Detection and Ranging (LiDAR) sensors. The downsides of both variants are an increase of costs, the necessity of involved recalibration routines and the inhibition of the product design form factors due to
fabrication constraints.

\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{teaser.png}
    \caption{
    Performance of state-of-the-art 3D detection methods on the KITTI3D validation and test sets\textcolor{red}{$^1$}. RGB-based methods (orange circles) exhibit a low performance discrepancy between the two sets, whereas Pseudo-LiDAR-based methods (green triangles)  perform much better (up to 10 AP) on validation than on test. This indicates a bias, which we display by means of a blue-toned colormap.
    These results also show that the best performing RGB-based methods generally benefit from exploiting a \textit{3D Confidence} (circled orange circles), a component which has not yet been introduced in any PL-based methods.
    }
    \label{fig:teaser}
    \vspace{-10pt}
\end{figure}

\footnotetext[1]{We took as reference the performance on class \textit{Car} in the \textit{Moderate} difficulty, computed with the $AP|_{R_{40}}$ metric \ie the one used as reference on the official KITTI3D benchmark.}

To overcome
these issues, an emerging branch of 3D object detection methods is entirely based on monocular cameras \cite{brazil2019m3d, jorgensen2019ss3d, ma2019am3d, Manhardt_2019_CVPR, simo2019monodis, simo2020virtual, wang2019pseudo}. Monocular cameras are a cheap alternative to the expensive LiDAR or stereo setups, but, at the same time, incur a substantially increased algorithmic complexity due to the absence of depth observations. Indeed, accurate estimation of an objects' distance to the camera is the most difficult task in monocular, image-based 3D object detection,
making it an ill-posed problem.
Despite the development of methods which focus on increasing the generalization with respect to distance \cite{brazil2019m3d, simo2020virtual}, monocular image-based methods still lag far behind their LiDAR or stereo-based counterparts.

A recent line of works~\cite{Manhardt_2019_CVPR,Xu_2018_CVPR} has leveraged Convolutional Neural Networks (CNNs) for image-based depth predictions as depth substitute in monocular 3D object detection algorithms. Pseudo-LiDAR (PL\xspace)~\cite{wang2019pseudo, yang2019pseudopp} was promoted as a particularly effective depth representation, reporting impressive results on the challenging KITTI3D benchmark \cite{Geiger2012CVPR}. It essentially mimics a LiDAR signal for a RGB image by projecting each 2D pixel from its corresponding, estimated depth map into 3D space. With the resulting 3D point cloud, the 3D detection task is usually approached by applying {state-of-the-art} LiDAR-based (and thus 3D point-based) detection algorithms. PatchNet~\cite{Ma_2020_rethink} has recently refuted 3D points as the source of PL\xspace's effectiveness by providing an equivalently performing implementation based on stacking 3D world coordinates as 2D maps. While this eliminates the claims of PL\xspace being advantageous due to its 3D point-based representation, their ablations confirmed the importance of operating on transformed 2D image coordinates incorporating camera intrinsics (focal length and principal point).

In this paper we argue that PL\xspace-based approaches, and more in general approaches that take depth as input, have introduced a distorted perception in the research community about their performance in the monocular setting with respect to other state-of-the-art methods that use RGB-images only. We identified two main reasons behind the issue, which constitute the two main contributions of this paper.

\myparagraph{First contribution.}
State of the art PL\xspace-based methods report excellent performance on the KITTI3D validation set but do not show the same gains on the test set. In this work we perform an in-depth experimental study to analyze the reasons behind such inconsistency and demonstrate that top performing PL\xspace-based methods adopt a training protocol which artificially leads to high average precision on the validation set. The issue is evident in Fig.~\ref{fig:teaser}, where the discrepancy between the KITTI3D validation and test set performance of PL\xspace-based methods (green triangles) is much more pronounced than RGB-based methods (orange circles).
Indeed, the depth estimation algorithms on which PL\xspace-based methods heavily rely are usually trained by including $\approx30\%$ of the validation set data used for 3D object detection. Despite this issue was mentioned briefly in~\cite{wang2019pseudo}, this biased training protocol was later used in many subsequent PL\xspace-based methods. This clearly indicates the necessity and the relevance for the community of a more detailed analysis, which we provide in this paper.

\myparagraph{Second contribution.} The outcome of a fair comparison on test set of PL\xspace-based methods against RGB-only based approaches
on the KITTI3D benchmark is currently favouring more the latter ones. On the flip side, we found that published PL\xspace-based methods are penalized by the complete lack of a proper 3D confidence score which, as shown in Fig.~\ref{fig:teaser} (circled orange circles), is becoming a fundamental component of state-of-the-art RGB-only methods.
In this paper we propose, for the first time, to endow PL\xspace-based methods with a mechanism for predicting a 3D confidence, demonstrating remarkable performance gains.
In particular, we show that, following previous RGB-only based methods \cite{simo2019monodis}, also in the case of PL\xspace, 3D confidence can be trained by directly regressing the expected loss. While this works well in practice, it is sensitive to the scale of the loss and, hence, requires some hyperparameter tuning. Moreover, it suffers from the issue of becoming overconfident as the training progresses towards an overfitting regime. In the spirit of addressing those two issues, we open a novel direction and successfully explore the possibility of having 3D confidences expressed in relative terms. Our novel finding leads to improved performance and sets the new state-of-the-art on the KITTI3D benchmark.

\section{Conclusions}
In this paper we have shown that top-performing Pseudo-LiDAR-based works suffer from a bias in the reported validation scores for the KITTI3D benchmark. The source of the issue is partially due to an overlap that exists between the training set used to train the upstream depth estimators, providing the depth in input to the PL\xspace-based methods, and the validation set used for 3D object detection.
In an attempt to validate the hypothesis we constructed an geographically separated training set for the depth estimators by ensuring geographical separation to the detection validation set. However we found that this is not sufficient to remove the bias in the validation set, which indicates the existence of a more structured nature of the issue. As a consequence, future works involving PL\xspace-based methods on KITTI3D should avoid comparative analysis against other methods using the validation set, but rather rely on the test set.
In the second part of our work, we provided an architectural change to PL\xspace-based methods aimed at endowing them with the ability of predicting 3D confidences. We showed that with this simple change PL\xspace-based methods get remarkable improvements on the KITTI3D benchmark establishing a new state of the art.

\section{Acknowledgements}
We would like to thank Andreas Geiger for supporting our experiments on the KITTI3D benchmark.
We also thank Xinzhu Ma, Wanli Ouyang and Garrick Brazil for sharing their detections and for helpful discussions.

{\small
\bibliographystyle{ieee_fullname}

\section{Related Works}

Current approaches for monocular 3D object detection can be roughly divided in two categories: RGB-only methods, which directly address the ill-posed problem of the object's distance estimation, and PL\xspace-based methods, which leverage from automatically estimated depth maps or point clouds to recover the distance information.

\myparagraph{Monocular RGB-only 3D detectors.} Earlier approaches for monocular RGB-only 3D detection such as SSD-6D~\cite{Kehl_2017_ICCV} and Deep3DBox~\cite{Mousavian_2017_CVPR} build on top of state of the art deep architectures for 2D detection, and exploit information from projective geometry to estimate the 3D pose and position of the objects in the scene.
Mono3D~\cite{Chen_2016_CVPR} develops from the idea of generating 3D proposals and scoring them according to several cues, such as semantic segmentation features, object contour, and location priors.
OFTNet~\cite{Roddick18} operates by considering an orthographic feature transform to map a 2D feature map to bird-eye view.
MonoGRNet~\cite{qin2019monogrnet}
simultaneously estimates 2D bounding boxes, instance depth, 3D location of objects and local corners.
GS3D~\cite{li2019gs3d} exploits an off-the-shelf 2D object detector and efficiently computes a coarse cuboid for each predicted 2D box, which is then refined to estimate the 3D bounding box. MonoPSR~\cite{ku2019monopsr} jointly leverages 3D proposals and scale and shape estimation to accurately predict 3D bounding boxes from 2D ones. Recently, few works have proposed single-stage deep architectures \cite{brazil2019m3d, simo2020virtual}. M3D-RPN \cite{brazil2019m3d} generates 2D and 3D object proposals simultaneously and exploits a post-processing optimisation and a depth-aware network to improve localization accuracy. MoVi-3D \cite{simo2020virtual} is a lightweight architecture which exploits automatically generated virtual views where the object appearance is normalized with respect to distance to facilitate the detection task.
Liu \etal~\cite{liu2020smoke} propose SMOKE, a deep architectures which predicts 3D bounding boxes by relying on key-point estimation as an intermediate task. MonoDIS~\cite{simo2019monodis} shows that training convergence and detection accuracy of 3D detection networks can be improved by considering loss disentanglement. In~\cite{simo2019monodis} 3D confidence for detection is also introduced for increasing performance. In this paper we show how this notion can be extended to PL\xspace and further improved introducing a relative measure of confidence.

\myparagraph{Pseudo-LiDAR based 3D detectors.}
A second category of works exploit external data and network models to generate depth maps from the RGB input as an intermediate step for 3D detection.
For instance, ROI-10D~\cite{Manhardt_2019_CVPR} introduces a loss to minimize the misalignment of 3D bounding boxes and exploits depth maps inferred with SuperDepth~\cite{Pillai_2019_ICRA}. A disparity prediction module is considered in~\cite{Xu_2018_CVPR} and integrated into a network composed of two parts: one that generates 2D region proposals, and another that predicts 3D object location, size and orientation.
Pseudo-Lidar~\cite{wang2019pseudo} represents the first PL\xspace method, introducing the idea of interpreting depth maps as 3D point clouds which are then fed to state-of-the-art LiDAR-based 3D object detectors. In \cite{wang2019pseudo} the presence of a possible performance bias is also suggested but an in-depth experimental study is lacking.
Pseudo-Lidar++~\cite{yang2019pseudopp} improves the accuracy in the localisation of faraway objects by adapting a stereo network architecture and deriving a loss function for direct depth estimation.
AM3D \cite{ma2019am3d} proposes to integrate complementary RGB features into the PL\xspace pipeline and introduces a a specific module to map the 2D image data
to the 3D point cloud.
PatchNet \cite{Ma_2020_rethink} analyses the effect of depth data representation on performances and improves over previous PL\xspace models by integrating the 3D coordinates as additional channels of input data.
However, all these works lack a fundamental component of state-of-the-art RGB-only based detectors \cite{simo2019monodis} \ie the estimation of a 3D confidence.

\section{3D Confidence for PL\xspace-based Methods}

As described in the previous section, the performance of PL\xspace-based methods is deeply influenced by the upstream depth estimation task. We will now demonstrate that the estimation of the 3D confidence has an equally relevant role.

The 3D confidence can be thought as an estimate of the quality of the 3D detection which, as described in Sec.~\ref{sec:3d_detection}, has to be associated to each 3D bounding box. In datasets such as KITTI3D, this confidence takes an active role in the computation of the metrics (\eg Average Precision). In light of this fact we observed that existing Pseudo-LiDAR methods
do not perform the 3D confidence estimation in any way but rely on the class probability coming along with the 2D detections.
By doing so, the confidence adopted by current PL\xspace-based methods is actually agnostic to the quality of the 3D predictions and therefore not effective for the role it should take.
On top of this, as shown in Fig.~\ref{fig:detections}, we observed that 2D detectors are often too confident and therefore the need for a 3D confidence seems essential.
For this reason, we propose to endow PL\xspace-based methods with the ability of estimating the 3D confidence.

\subsection{Proposed Architecture}
In order to describe how the confidence is estimated we first provide an overview of the general architecture we adopt, similarly to other PL\xspace-based methods \cite{wang2019pseudo, Ma_2020_rethink}, and subsequently detail our contribution.

\myparagraph{PL-based 3D detection architecture.}
The architecture commonly adopted by state-of-the-art PL\xspace-based methods, which we also use in this work, is depicted in Fig.~\ref{fig:pipeline_all} (excluding the red block, \ie our contribution).
It can be divided into three main branches, namely \textit{2D Detection}, \textit{Pseudo-LiDAR} and \textit{3D Detection}.
The \textit{2D Detection} and \textit{Pseudo-LiDAR} components usually exploit pre-trained architectures and have the purpose of understanding where the object of interest are in the image, as well as of estimating the per-pixel depth, respectively. The per-pixel depth-map is then converted to Pseudo-LiDAR 3D point-cloud and, finally, the points belonging to each object are selected and filtered to discard elements corresponding to \eg road, occlusions.
The 3D detection block is responsible for the estimation of the output 3D bounding boxes, taking as input the selected PL\xspace points to perform a point-based 3D detection by means of an initial \textit{3D Backbone} followed by a \textit{3D Head}.

\myparagraph{3D confidence head.}\label{sec:conf_head} In the following we describe our main contribution, \textit{i.e.} an approach which endows the PL\xspace-based methods under consideration with the ability to predict a self-supervised 3D confidence.
In order to reliably and accurately estimate the 3D confidence of bounding boxes, appropriate 3D-related feature representation need to be computed. For this reason, in this work we introduce an additional branch in the architecture, namely the \textit{3D Confidence Branch}, which, as shown in Fig.~\ref{fig:pipeline_all} (red block), takes as input the set of \textit{3D Features} computed by the \textit{3D Backbone} and outputs a single value $C_i$, \ie the 3D confidence, for each object. In the presence of $K>1$ classes the output is a set of $K$ confidences $C_i^k$, one for each class $k$.
Note that our proposed \textit{3D Confidence Branch} is not tied to any particular architecture and requires minimal modifications to existing PL\xspace approaches. An example of a simple implementation is by mirroring the architecture of the \textit{3D Head}, thus leading to limited computational complexity and minimal overhead in term of inference time.

\subsection{Learning the 3D Confidence}\label{sec:learn_conf}
In this paper we propose two different approaches for 3D confidence prediction. The first approach, which we denote as \textit{Absolute 3D confidence} estimation is inspired by previous RGB-only based methods \cite{simo2020pami}, while the second strategy, called the \textit{Relative 3D confidence} estimation method, is introduced with this paper. In both cases, given a 3D bounding box $B_i$ and the corresponding ground-truth $\hat B_i$, the loss for the 3D confidence prediction $C_i^\text{3D}$ takes the following cross-entropy form:
\[
L_\text{conf}(C^\text{3D}_i|B_i,\hat B_i)=-T_i\log C^\text{3D}_i - (1-T_i)\log(1-C^\text{3D}_i),
\]
where $T_i$ is the target confidence value that takes a different value for the absolute and relative confidences, as described below.
In case of multiple object categories, we assume to have independent 3D confidence predictions per class.

\myparagraph{Absolute 3D confidence.}
Inspired by \cite{simo2019monodis}, the absolute 3D confidence is trained by directly regressing the loss of the prediction.
This boils down to setting
$$T^\text{abs}_i=e^{-\frac{1}{\beta}\ell(B_i,\hat B_i)}$$ as the target confidence, where $\ell(B_i,\hat B_i)$ is the loss incurred by the bounding box prediction and $\beta > 0$ is a temperature parameter.
Since this approach leads to a 3D confidence which reflects the quality of a 3D detection in absolute terms we call it \textit{Absolute 3D Confidence}.

\myparagraph{Relative 3D confidence.}
We also propose a novel approach which aims at overcoming a couple of issues that affect loss-based confidences like the one described above. The first is that they are sensitive to the scale of the loss values, which requires to tune scaling factors. The second is that they are not immune to the issue of the network becoming overconfident as the training progresses towards an overfitting regime.
We solve these two issues by shifting the semantics of the score that the network provides with each prediction from an absolute confidence to a relative confidence.
A typical confidence score is supposed to be representative of the absolute quality of the prediction. Instead, the score that we require the network to learn is representative of the quality of the prediction relative to other typical predictions done by the network. For this reason, this novel confidence is regarded as \textit{Relative 3D Confidence}.

\noindent Consider a training set containing $n$ 3D objects, where
$$\ell_i \triangleq \ell(B_i,\hat B_i)$$ denotes the loss incurred by the model on the predicted 3D bounding box for the $i$th object.
We propose to regress as a confidence $C_i^\text{3D}$ for the prediction $B_i$, the proportion of 3D objects in the training set on which the model performs equal or worse than on object $i$, \ie our target confidence prediction is given by
\[
T^\text{rel}_i=\frac{1}{n-1}\sum_{j=1\atop j\neq i}^n {\mathbb 1}_{\ell_j\geq\ell_i}\,,
\]
where $\mathbb 1_P$ denotes the indicator vector for predicate $P$.
The proposed confidence is inherently relative because it does not depend on the actual absolute values of the losses, but rather on their ordering.
In order to train a model to regress the new confidence, we need to compute the value of $T^\text{rel}_i$ for each 3D object $i$ in the mini-batch. However, to compute such target values, we need to access the loss incurred on each 3D object in the training set, which is computationally demanding. An alternative solution consists in tracking the past loss values for each 3D object in the training set. This is feasible, but given the frequent updates of the model the past losses would be soon outdated. Also, there exist augmentation strategies, for which it might be hard to match predictions across epochs. Interestingly, there is a very simple, stochastic procedure that allows us to train the desired confidence by only accessing loss values in a mini-batch of at least $2$ elements. Specifically,
we randomly pair the bounding box prediction for 3D object $i$ in the mini-batch with the prediction obtained for another, distinct 3D object $\pi_i$ in the same set.
Given this assignment, we compute a binary target value $\hat T_i$ for training the confidence as
$ \hat{T_i} = \mathbb{1}_{ \ell_i \leq \ell_{\pi_{i}}},$
and plug this into the cross-entropy loss $L_\text{conf}$ that we use to train the confidence. To see why this works, fix a 3D object $i$ in the training set. Then the target variable $\hat T_i$ is a random variable, where $\pi_i$ is uniformly sampled among the other $n-1$ 3D objects in the training set. Accordingly, the expected value $\mathbb E[\hat T_i]$ of $\hat T_i$ yields exactly $T^\text{rel}_i$. This is also the value to which the predicted confidence $C_i^\text{3D}$ will tend to if trained with the cross-entropy loss $L_\text{conf}$, assuming the losses will eventually converge during training.
If multiple classes are present, we have independent confidence predictions per class and the random pairing procedure is applied only between 3D objects in the mini-batch having the same class. No loss is computed if a 3D object is the only one of a given class in the mini-batch.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.99\columnwidth]{err_bands.pdf}
    \caption{Binned scatter plot of the \textit{Absolute} (left) and \textit{Relative} (right) 3D confidences on the training (green) and validation (red) splits. Since the x-axis represents what the confidence is supposed to regress, the ideal curve lies on the diagonal. On the training set, both the absolute and relative confidences follow the expected curve. On the validation set however only the relative confidence aligns with the expected curve, whereas the absolute confidence consistently overestimates it, signaling the issue of being overconfident.}
\label{fig:err_bands}
\end{figure}

\myparagraph{Pros and cons of relative confidences.}
Our score takes the role of a \emph{relative} confidence, which does not give information about the quality of a prediction in absolute terms, but rather assesses the quality of the prediction relative to other predictions done by the network.
As an extreme example, a nearly perfect prediction can get zero confidence if that is the prediction that incurred the highest error in the training set. Similarly, a bad prediction can get high confidence score if it is the best prediction the network ever made. An interesting property of our confidence score is the invariance to order-preserving transformations of the losses. This renders the score more robust to scenarios where the losses change over time, which is the setting that is encountered at training time. Additionally, our confidence score does not suffer from the problem of the network becoming overconfident, because of the relative nature of our score (see Fig.~\ref{fig:err_bands}).
On the downside, an absolute confidence is typically helpful to filter predictions based on a threshold. Doing the same with a relative confidence might be cumbersome.
This is why we actually combine our relative confidence score from the 3D detection head with the 2D \emph{absolute} confidence score from the 2D detection. Indeed, the 2D detection confidence works well for the sake of removing bad quality predictions, but lacks resolution for discriminating the quality of the predictions left. This is where our relative confidence comes into play, since it does not suffer from the issue of becoming overconfident.
\section{Additional details about GeoSep}
As explained in Sec.~\textcolor{red}{4.2} of the main paper, we create novel training and validation depth splits (\textit{GeoSep)} by introducing geographical separation between the commonly used depth training set (\textit{Eigen~et~al.}~[\textcolor{green}{6}]) and the object detection dataset (KITTI3D).

To create the \textit{GeoSep} splits we exploit the GPS information included in the available KITTI Benchmark data, and define two separation criteria. We withhold all images i) captured closer than 200m from any KITTI3D training or validation detection image, and ii) belonging to any of the KITTI3D detection sequences.
From a total available amount of 47962 images the aforementioned filtering process yields 22954 images, which we divide in 22287 for the training set and 667 images for the validation set. The distance threshold has been chosen to ensure that our novel \textit{GeoSep} splits would have approximately the same number of images of the \textit{Eigen~et~al.}~[\textcolor{green}{6}] splits, which are 23488 for training and 697 for validation, respectively.

\section{Additional implementation details}
In this section we provide additional details about the implementation and additional information about the hyper-parameters. Since our method is subdivided into multiple branches, we provide details of each one namely \textit{2D Detection}, \textit{Pseudo-LiDAR} and \textit{3D Detection}. In all our experiments, we trained our models on a single NVIDIA GTX 1080 Ti with 11GB of memory.

\vspace{-10pt}
\paragraph{2D Detection.}
As described in Sec.~\textcolor{red}{6} of the main paper we do not train a 2D detector but instead rely on pre-computed 2D detections. In our experiments we used, for both validation and test set, the 2D detections used in PatchNet~[\textcolor{green}{18}].

\vspace{-10pt}
\paragraph{Pseudo-LiDAR.}
We took the open-source code of BTS~[\textcolor{green}{12}] and selected the DenseNet161-based estimator. For our results on the Eigen et al.~[\textcolor{green}{6}] we used the model trained by the authors\footnote{\url{https://cogaplex-bts.s3.ap-northeast-2.amazonaws.com/bts_eigen_v2_pytorch_densenet161.zip}}. For the trainings on our our novel \textit{GeoSep} splits, we used the ImageNet~[\textcolor{green}{32}] pre-trained model and followed the official schedule and hyperparameters.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\columnwidth]{pix/pipeline_3d.pdf}
    \caption{Example of final part of the 3d detection architecture, where we introduced our proposed \textit{3D Confidence Head} in PatchNet~[\textcolor{green}{18}]. The implementation of the \textit{3D Confidence Head} (red) follows the one of the \textit{3D Head} (blue).}
    \label{fig:3d_conf_head}
    \vspace{-12pt}
\end{figure}

\vspace{-10pt}
\paragraph{3D Detection.}
The architecture of our proposed models, \ie the ones based on Wang et al.~[\textcolor{green}{28}] and PatchNet~[\textcolor{green}{18}], always follow the official one with the only exception of the introduction of our proposed \textit{3D Confidence Head}. The implementation of this particular head closely follows the one of the respective 3D Head. In particular, for our implementation based on Wang et al.~[\textcolor{green}{28}] we introduced a series of three fully-connected layers with \textit{512-D}, \textit{512-D}, and \textit{1-D} dimensions respectively. For the implementation of PatchNet we introduced three distance-specific heads composed by a series of three fully-connected layers with \textit{512-D}, \textit{512-D}, and \textit{1-D} dimensions respectively. We depict the PatchNet \textit{3D Head} (blue), along with our proposed \textit{3D Confidence Head} (red), in Fig.~\ref{fig:3d_conf_head}. We trained our model with the Adam optimizer with a learning rate of 0.001 and a batch size of 64 for 100 epochs, decreasing the learning rate by a factor of 0.1 at the $20^{th}$ and $40^{th}$ epoch.

\section{Additional qualitative results}
After a initial description of the visualization method, we provide additional qualitative results of our detections on KITTI3D.

\vspace{-10pt}
\paragraph{Visualization method.}
We visualize our results by super-imposing our \textit{PatchNet + Relative 3D Confidence} 3d bounding box detections on the input RGB image as well as rendered Pseudo-LiDAR pointcloud, as presented in Fig.~\ref{fig:video}.

\begin{figure}[h!]
    \centering
    \resizebox{0.99\columnwidth}{!}{
        \includegraphics[width=\textwidth]{pix/video.png}
    }
    \caption{Example of output visualization. Top: Visualization of our predictions on the colored Pseudo-LiDAR pointcloud. Bottom: Visualization of our predictions, with corresponding confidence score, on the input RGB image.}
\label{fig:video}
\end{figure}

In the top part of the images we visualize our detections on the rendered Pseudo-LiDAR pointcloud, where each point has been colored with its corresponding RGB value (if available), and consequently visualized our predicted 3d bounding-boxes in green for \textit{Car}, cyan for \textit{Cyclist} and red for \textit{Pedestrian}. The presence of black pixels (\eg on top of objects) is due to the fact that we rendered the scene from a point-of-view which is different from the one of the KITTI3D RGB camera. This change of pose inevitably introduces these black pixels on regions which were not visible from the RGB camera pose.

\vspace{-10pt}
\paragraph{Qualitative results on images.}
In Fig.~\ref{fig:ablation},\ref{fig:ablation2} we show our results on KITTI3D test set images. Our proposed confidence is demonstrated to reliably determine the overall quality of the predicted 3D bounding box. The confidence is in fact higher on nearer and not occluded objects, \ie where the estimation is more reliable, and seems to degrade with distance and occlusions. We also included some failure cases in which our confidence is shown to be less reliable. In particular, we have identified some imprecise or empty detections that still have fairly high confidence.

\paragraph{Qualitative results on video sequences.}
We further provide a qualitative video (\textit{3721.mp4}) by showing our predictions on complete KITTI3D sequences taken from the KITTI3D validation set. Unfortunately, it was not possible to provide videos on the KITTI3D test set sequences due to the unavailability of test set sequence information. The predictions are computed for each frame in an independent manner, without exploiting temporal information in any way. Despite the presence of failure cases, \eg when object are too near/far/occluded, our confidence score is shown to generally reflect the quality of the 3d detection.

\section*{References}
\small
\begin{itemize}
\item [{[32]}] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. \textcolor{red}{1}
\end{itemize}

\iffalse
\begin{figure*}[h]
    \centering
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/003778.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000106.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/006724.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/001792.jpg}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation}
    \vspace{-12pt}
\end{figure*}
\begin{figure*}[h]
    \centering
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000075.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000119.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/001527.jpg}
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000108.jpg}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation2}
    \vspace{-12pt}
\end{figure*}
\fi

\begin{figure*}[h]
    \centering
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000026.png}
    \\ ~ \\
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000178.png}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation}
    \vspace{-12pt}
\end{figure*}
\begin{figure*}[h]
    \centering
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/000175.png}
    \\ ~ \\
    \includegraphics[width=2.0\columnwidth]{pix/pix_supp/002784.png}
    \caption{Additional qualitative results of our 3d bounding box detections on the KITTI3D test set.}
    \label{fig:ablation2}
    \vspace{-12pt}
\end{figure*}

\end{document}


text:
\section{Introduction}\label{sec:intro}

In 1935, Albert Einstein, Boris Podolsky and Nathan Rosen published a paper
that sent shock waves in the physics community~\cite{epr35}, especially in Copenhagen.
L\'eon Rosenfeld wrote about the impact this paper had on Niels Bohr,
one of the fathers of quantum mechanics and its hardiest
defender at the time, that
``this onslaught came down to us as a bolt from the blue''~\cite{bolt}.
Quantum mechanics (QM) was still a relatively young theory, which had
arisen from several experimental facts.
It~exhibited so many weird features that Bohr once said:
``Anyone who is not shocked by quantum theory has not understood~it''\,%
\footnote{\,Or something of the kind: we can find a variety of similar ``quotes''
attributed to Bohr and it's not clear what he really said!
See~\url{http://en.wikiquote.org/wiki/Niels_Bohr}.}.
Among the non-classical effects of QM, we note its fundamentally
probabilistic nature and the fact that we cannot determine simultaneously
the position and momentum of a particle with arbitrary precision.
In~fact, forget about \textit{measuring} them: QM~tells us that these two conjugate variables
cannot have simultaneous \textit{existence}. These exotic features did not
appeal to everyone in the physics community. To~some, they were an indication that
QM was incapable of fully describing physical reality.
So, with this thought in mind,
Einstein, Podolsky and Rosen (EPR) published the celebrated paper entitled
``Can~quantum-mechanical description of physical reality be
considered complete?''~\cite{epr35}, in which they put forth an argu\-men\-tation
purportedly answering their own question in the negative.
Seventy years after its publication, this is \emph{still} the most heavily
downloaded paper from the American Physical Society website
\url{www.aps.org}\,!

Even though there is evidence~\cite{chris,jammer} that Einstein had been thinking about
it as early as 1931, the notion of \emph{entanglement} was first published in
the 1935 EPR paper. The authors used the correlations obtained from bipartite
measurements of an entan\-gled state to claim that it is possible for the
position and momentum of a particle to have simultaneous existence.
Since the formalism of QM precludes such a possibility,
they concluded that QM cannot offer a complete description of physical reality.
Even though the term ``local hidden variable'' (LHV) was yet
to be coined, it follows in their opinion that nature has to feature
LHVs that cannot be known collectively through any experiment---for~otherwise
the Heisenberg uncertainty relations would be violated---but~never\-theless
these variables determine the behaviour of the physical system
under measurement. Einstein then spent much of the rest of
his life in the vain search of such a theory.

Almost thirty years elapsed before a paper by John~S.
Bell doomed EPR's quest for a LHV theory of nature~\cite{bell64}.
More specifically, Bell defined the most general LHV model possible
and showed that no such model can yield the correlations
predicted by QM when the entangled system proposed by EPR---or~more
precisely its variation subsequently proposed by
David Bohm~\cite{bohm51}---is~measured,
proving in effect that the EPR argument could \emph{not} show that QM is
incomplete. Nevertheless, the situation could have been worse
for quantum mechanics: it~could have been outright incorrect!
Subsequent experi\-ments~\cite{clauser,aspect} established
the supremacy of QM over LHV theories,
a~somewhat ironic backfiring of the intent of the EPR paper.
It~is a great pity that Einstein did not live to see these breakthroughs.

Even though it took the better part of three decades before Bell
was able to reject all possible LHV models, EPR's conclusions
were debated right away. Two months after the publication of the
EPR paper, Bohr submitted a response
to the same
journal, with the exact same title~\cite{bohr35}.
Quite obviously, Einstein was left entirely unconvinced by Bohr's response.
More on this in Section~\ref{bohr}.

We wish to present here, in loving memory of Asher Peres, a
hopefully insightful \mbox{discussion} of the EPR argument.
The originality of our approach, from a contemporary perspective, is that
its purpose is to poke holes in the the EPR paper \emph{without any
reference to Bell's \mbox{inequality}}. Consequently, its subtitle could have been:
``Why~we would have questioned EPR independently of Bohr's response,
even before Bell came along''!
The~holes we poke are nowhere as fatal as Bell's and they do \emph{not}
rule out the possibility of a LHV theory, which was therefore still a legitimate
field of research until the fateful year~1964.
Instead, we show that the EPR exposition of their ideas
needed polishing for it is marred by several \emph{logical} errors.

After this Introduction, we summarize in Section~\ref{sec:epr}
the EPR argument by quoting the
most relevant excerpts from their paper.
Section~\ref{logic} is the main contribution of our paper:
we~rephrase those quotes in the language
of mathematical logic, which allows us to analyse precisely the
meaning of the words used by~EPR, and exhibit logical holes
in them.
We~address the issue of \emph{spukhafte Fernwirkungen} in
Section~\ref{epistemic}.
In~Section~\ref{bohr}, we present an overview of Bohr's
response to the EPR paper in the interest
of showing how it failed to address the gist of the EPR argument.
Finally, we conclude in Section~\ref{sec:conclusion}.

\section{The Einstein-Podolsky-Rosen Argument}\label{sec:epr}

As for what should be considered a valid theory of nature, EPR
proposed the following two criteria: (1)~the theory should be
correct and (2)~the theory should be complete. Of~course, the
scientific paradigm should also demand falsifiability as a criterion of
validity, but the EPR argument hinges upon the \emph{completeness}
(or~purported lack of completeness) of~QM\@.
Specifically, the authors give the following definition.\footnote{\,%
Throughout this paper, we set quotes in \textsf{sans serif} whenever
they are taken verbatim from the EPR paper~\cite{epr35} or Bohr's response~\cite{bohr35},
and we use (sans serif) italics in strict accordance with the quoted material.}

\begin{defi}[Completeness]\label{def:completeness}
\textsf{\textit{Every element of the physical reality must have a counterpart in
the physical theory.}}
\end{defi}

The definition of a \emph{physical reality} given by EPR, which
they ``\textsf{regard as reasonable}'' with no further apparent need for justification,
is spelt out in a very famous quote from their paper.

\begin{defi}[Physical reality]\label{def:reality}
\textsf{\textit{If, without in any way disturbing a system, we can predict with
certainty (i.e., with probability equal to unity) the value of a
physical quantity, then there exists an element of physical
reality corresponding to this physical quantity.}}
\end{defi}

As we shall see, the EPR argument is centred around two
statements.

\begin{stata}
Quantum mechanical description of reality is not complete.
\end{stata}

\begin{statb}
Non-commuting operators cannot have simultaneous reality.
\end{statb}

They use both of these statements in an intricate manner in order
to come up with the final ``conclusion'' that Statement~{\text{A}} must be true.
The~structure of their proof can be established from the
following two quotes, which appear at the end of the first section of their paper.

\begin{itemize}
\item[(a)]
\textsf{either (1) \emph{the quantum-mechanical
description of reality  given by the wave function is not
complete} or (2) \emph{when operators corresponding to two
physical quantities do not commute the two quantities cannot have
simultaneous reality}.}

\item[(b)]
\textsf{In quantum mechanics it is usually assumed that the wave function
\emph{does} contain a complete description of the physical reality
of the system in the state to which it corresponds.
[\ldots]
We~shall show, however, that this assumption, together with the
criterion of reality given above, leads to a contradiction.}

\end{itemize}

\noindent
The argument culminates in their conclusion:

\begin{itemize}
\item[(c)]
\textsf{Starting then with the assumption that the wave function does give
a complete description of the physical reality, we arrived at the
conclusion that two physical quantities, with noncommuting
operators, can have simultaneous reality.  [\ldots]
We~are thus forced to
conclude that the quantum-mechanical description of physical
reality given by wave functions is not complete.}
\end{itemize}

\section{The Language of Logic}\label{logic}

To extract the substance of the EPR argument, we shall now rephrase its structure
in the \mbox{language} of mathematical logic. In~this language, Quote~(a) translates to:
``Either Statement~{\text{A}} or Statement~{\text{B}}''.
The normal usage of ``\emph{either}~$p$ or $q$'' in
English corresponds mathematically to the \emph{exclusive-or} of $p$ and $q$,
namely that either $p$ or $q$ is true \emph{but not both}.
This is written \mbox{$p \oplus q$} in modern mathematical notation.
Therefore, EPR's first quote amounts to saying that
\begin{equation}\label{eq:claima}
\text{A} \oplus \text{B} \, .
\end{equation}

Quote~(b) is a little more subtle to translate into the language
of mathematical logic. First of all, we must understand that
``\textsf{the~criterion of reality given above}'', at the end of the quote,
corresponds to Statement~{\text{B}} being true.
Therefore, Quote~(b) translates to
\mbox{$\neg \text{A} \wedge \text{B} \Rightarrow \textsf{false}$}.
This is mathematically equivalent to saying that
\begin{equation}\label{eq:claimb}
\neg \text{A} \Rightarrow \neg \text{B}
\end{equation}
as can be verified easily using a truth table.
Proposition~(\ref{eq:claimb}) is stated more explicitly
in the first sentence of Quote~(c) above.

\newpage

Given that EPR eventually derive their desired conclusion,
namely Statement~{\text{A}}, \mbox{after} ``estab\-lishing'' the truth
of both quotes, their final argument---Quote~(c)---hinges upon logical \mbox{tautology}
\mbox{$(\ref{eq:claima}) \wedge (\ref{eq:claimb}) \Rightarrow \text{A}$}:
\begin{equation}\label{eq:tautology}
(\text{A} \oplus \text{B}) \wedge (\neg \text{A} \Rightarrow \neg \text{B}) \Rightarrow \text{A} \, .
\end{equation}

Having set the stage for our logical analysis of the EPR argument,
we now proceed to discussing these three propositions one by~one.

\subsection{The First Proposition}\label{sc:first}

In order to establish $\text{A} \oplus \text{B}$, it must be shown that either
Statement~{\text{A}} or Statement~{\text{B}} holds. To~do this, EPR
used the fact that the formalism of QM does not allow for
a particle's position and momentum to be simultaneously defined.
Therefore, if position and momentum do indeed have
simultaneous reality (negation of Statement~{\text{B}}) then QM
cannot be complete (Statement~{\text{A}}). In~other words,
\mbox{$\neg \text{B} \Rightarrow \text{A}$}. Furthermore, they also
argued that  if QM is already a complete theory
(negation of Statement~{\text{A}}) then it must be the case
that position and momentum cannot have
simultaneous reality (Statement~{\text{B}}). In~other words,
\mbox{$\neg \text{A} \Rightarrow \text{B}$}.

At this point, we remark that working through
both implications was redundant since they are logically equivalent,
each one being the contrapositive of the other.
A~more serious problem with the EPR argumentation is that the
desired conclusion~(1) does \emph{not} follow!
Indeed, \mbox{$\neg \text{A} \Rightarrow \text{B}$} is logically equivalent
to \mbox{$\text{A} \vee \text{B}$}, which cannot be used to conclude
that \mbox{$\text{A} \oplus \text{B}$}.
Worse: no amount of logical reasoning can save the day
since nowhere do EPR argue that it's not possible for
both Statements~{\text{A}} and~{\text{B}} to hold, and indeed that would
be possible should QM be incomplete for some other reason!

We suspect that this problem with the EPR argument simply comes from
our (linguistically correct) interpretation of the word ``either''
in Quote~(a) from Section~\ref{sec:epr}: Instead of Proposition~(\ref{eq:claima}),
no doubt EPR had
\begin{equation}\label{eq:claimabis}
\text{A} \vee \text{B}
\end{equation}
in mind, which indeed is correct, even by today's account.
This happened \emph{either} because EPR were not careful in their use
of the English language, or because the meaning of ``either'' has evolved
in the past seventy years.
But~there is no reason to worry further about this issue since we shall see
in Section~\ref{sc:tautology} that it has no real impact on the EPR conclusion.

\subsection{The Second Proposition}

In order to ``prove'' $\neg \text{A} \Rightarrow \neg \text{B}$,
Einstein, Podolsky and Rosen
used a bipartite entangled state of two particles, where two physicists,
say Alice and Bob, take one particle each and move to space-like
separated regions. Let~us assume that Alice performs a
momentum measurement on her particle. From the measurement
postulate of QM, she is then able to predict with certainty the
outcome of a momentum measurement on Bob's particle. Since Alice and
Bob are in space-like separated regions, there cannot be any
further interaction between the particles once they are separated.
Therefore, Alice's measurement cannot have disturbed Bob's system
in any way. Hence, according to Definition~\ref{def:reality},
momentum corresponds to an element of physical reality for Bob's particle.
Let us change the
assumption to a position measurement on Alice's part of the
entangled state, \emph{instead of} a momentum measurement.
Once again, she will be able to predict the result of a
position measurement on Bob's particle without having disturbed~it.
Therefore, position also corre\-sponds to an element of
physical reality for Bob's particle.
It~``follows'' that position and momentum (two non-commuting operators) had to
have simultaneous reality before either measurement.
This ``establishes'' that \text{B}\ is~false.

The flaw, of course, is that
EPR's argument hinged upon counterfactual arguments of the kind
``the~physicist could do this \emph{or} he could do that'' in cases when
clearly he could not do both.
Nobody put this flaw in better light than Asher Peres, in his
famous aphorism:
``Unperformed experiments have no results''~\cite{noresults}.
Since Alice cannot perform both measurements, it is not the case
that both outcomes are defined simultaneously.
Being fully aware that Einstein would have been left utterly unconvinced
by this reasoning, we take here a different line of attack,
based on logic rather than physics.

Recall that we were supposed to prove~$\neg \text{A} \Rightarrow \neg \text{B}$.
The careful reader will have noticed that EPR reached their desired~``$\neg \text{B}$''
conclusion \emph{without having at any point invoked the assumption}~``$\neg \text{A}$''\@!
Indeed, the reasoning did \emph{not} hinge upon the completeness of QM or lack thereof.
Instead, it hinged upon the \emph{correctness} of~QM, in particular the correctness
of the measurement postulate. Therefore, if we are willing to assume that QM is
correct in terms of its predictions about what can be observed---which we believed
to have been EPR's point of~view---then the EPR argument that purports to prove
\mbox{$\neg \text{A} \Rightarrow \neg \text{B}$} in fact directly ``proves''~$\neg \text{B}$.

\subsection{The Tautology}\label{sc:tautology}

{From} a purely logical point of view, there is nothing to complain about here\ldots\ at~least
on first approach.  Proposition~(\ref{eq:tautology}) is indeed a tautology, as is easily seen
by use of a truth table or a rather simple logical argument\,%
\footnote{\,Assume~\text{A}\ to be false. Then~\text{B}\ is true since $\text{A} \oplus \text{B}$.
But then \text{A}\ would be true since \mbox{$\neg\text{A}\ \Rightarrow \neg\text{B}$} is equivalent to \mbox{$\text{B} \Rightarrow \text{A}$} and \text{B}\ is true.
This contradicts the assumption that \text{A}\ is false, and it must therefore be the case that \text{A}\ is true.}.
However, this tautology is not genuinely relevant to the EPR argument.
First of all, as discussed in Section~\ref{sc:first}, Proposition~(\ref{eq:claima}) should be
replaced by Proposition~(\ref{eq:claimabis}) in the tautology~(\ref{eq:tautology}), yielding
\begin{equation}\label{eq:tautologybis}
(\text{A} \vee \text{B}) \wedge (\neg \text{A} \Rightarrow \neg \text{B}) \Rightarrow \text{A} \, .
\end{equation}
The good news is that Proposition~(\ref{eq:tautologybis}) is also a tautology,
whence the issue of the precise meaning of ``either\ldots or'' turns out to
be moot, as promised at the end of Section~\ref{sc:first}.

But now remember that, notwithstanding Quote~(b), the reasoning given by EPR does \emph{not} prove Proposition~(\ref{eq:claimb})
since they used nowhere the assump\-tion ``\mbox{$\neg\text{A}$}'' that QM is complete. Rather, they ``proved'' \mbox{$\neg\text{B}$}
directly, under the sole assumption that QM is \emph{correct}---in~particular the measurement postulate---which
seems to be taken for granted throughout the paper so strongly that it's not worth calling it an ``assumption''.
Therefore, the somewhat complicated-looking tautology that we wrote down as Proposition~(\ref{eq:tautology}),
to translate the English written words of~EPR, boils down in the end to nothing more than:
\begin{equation}\label{eq:triviality}
(\text{A} \vee \text{B}) \wedge \neg \text{B} \Rightarrow \text{A} \, .
\end{equation}
This statement is such a triviality\,%
\footnote{\,If~{\text{A}} or {\text{B}} is true and it is not {\text{B}}, then it must be~{\text{A}}.
Please compare this with the reasoning spelt out in the previous footnote!}
that we are forced to conclude that the original tautology was a smoke-screen at best.

In summary, the EPR conclusion may have been correct had they been allowed to make counterfactual reasoning,
but their logic to get there was unnecessarily intricate and even outright wrong in some places.

\newpage
\section{Spukhafte Fernwirkungen}\label{epistemic}

Consider the following quote from the EPR paper, which corresponds to the situation after
an interaction has left two particles in the singlet state.

\begin{quote}\sf
We can then calculate with the help of Schr\"odinger's equation the state
of the combined system [\ldots].
We~cannot, however, calculate the state in which either one of the
two systems is left after the interaction. This, according to
quantum mechanics, can be done only with the help of further
measurements, by a process know as the \emph{reduction of the wave
packet}.
\end{quote}

\noindent
{From} a modern perspective, this is utter nonsense!
\emph{Of~course} we can calculate the state of either system by use of
partial tracing: it's the completely mixed state.
Obviously, EPR considered that only \emph{pure} states deserve being
called ``states''.
Given their insistence on ``elements of reality'', this position
is hardly surprising.
Nevertheless, we must point out that von Neumann had already introduced
density matrices~\cite{vonneumann32}
a few years before EPR wrote their~paper.

If we take the epistemic viewpoint according to which the density matrix
corresponding to Bob's possible knowledge is all there is to the state of his particle,
then Alice's measurement in a space-like separated region has no instantaneous
influence on that state.
It's only if and when Alice tells Bob the result of her measurement that
the state of Bob's particle changes \emph{at~Bob's}, but this communication cannot
take place faster than the speed of light: \emph{spukhafte Fernwirkungen} no more!\,%
\footnote{\,With apologies to Spider-Man! \texttt{:-)}}

We mentioned the above in the interest of readers who may still be under the delusion
that Alice's measurement has an instantaneous influence on the state of Bob's particle.
However, we acknowledge that this issue is not central to a discussion of the EPR paper
since they took the unambiguous (and correct) position that such influence would not take place.
Their mistake was in taking the ontic position that all states are pure, from which it follows that
the state of Bob's particle is defined independently of Alice's or Bob's knowledge about it.

\section{Bohr's Response}\label{bohr}

In his response, Bohr~\cite{bohr35} hardly addresses the issue of entanglement.
For him, the phenomenon described in the EPR paper is nothing more
than what he calls \emph{complementarity}. He~claims that in~QM, as well
as in general relativity, we must take into account the measuring
device to make predictions on the object of interest. In~essence,
Bohr argues that position and momentum cannot both be measured
with an arbitrary precision
\emph{because we cannot do so experimentally}, even in principle.
A~measure of the momentum of a particle can only
be experimentally realized by a momentum transfer on the
measurement apparatus, therefore creating a displacement and
forsaking forever the knowledge of the position of the particle.
In~the EPR argument, says Bohr, when Alice measures the momentum of her
particle, she learns that of Bob's, but at the same time
renounces by her own free choice to the
knowledge of position that she could have obtained on Bob's particle.
This renunciation is based
on the fact that we cannot predict the result of a measurement.
In~the example given above, if we want to measure the momentum of a
particle, since there is no way to predict either the momen\-tum
exchange or the displacement caused by it, we cannot know with
certainty where the particle interacted with the apparatus.
The following quote summarizes Bohr's argument.

\begin{quote}\sf
In fact, the renunciation in each experimental arrangement of the
one or the other of two aspects of the description of physical phenomena,---the
combination of which characterizes the method of classical
physics, and which therefore in this sense may be considered as
\emph{complementary} to one another,---depends essentially on the
impossibility, in the field of quantum theory, of accurately
controlling the reaction of the object on the measuring
instruments, i.e.,~the transfer of momentum in case of position
measurements, and the displacement in case of momentum
measurements.
\end{quote}

The problem, of course, is that EPR had no issue with comple\-men\-tarity!
They did not claim in their paper that it would be possible to
\emph{know} simultaneously the position and
momentum of a particle with arbitrary precision.
Indeed, had they not agreed with this manifestation of
Heisenberg's uncertainty relations, they would surely have claimed
that QM is outright \emph{incorrect}, not merely incomplete!
Their disagreement with the view of QM is summarized in the following
quote from their paper:

\begin{quote}\sf
The usual conclusion from this\,%
\footnote{\,Measuring the coordinate of a particle alters its state.}
in quantum mechanics is that
\emph{when the momentum of a particle is known, its coordinate
has no physical reality}.
\end{quote}

\noindent
So, we see that, in the views of EPR, the incompleteness of QM has to do with the fact
that it does not allow simultaneous \emph{reality} for two non-commuting
observables. Whether or not these two realities could be learned
simultaneously is irrelevant to their argument.
The fact that QM allows us to predict
either the momentum or the position of a particle without having interacted with it,
mutually exclusive as these
measurements might be, seems to mean, according to
Definition~\ref{def:reality}, that both of these quantities have
physical reality. Therefore QM appears to be incomplete, according to their
criterion of Definition~\ref{def:completeness}.

\section{Conclusion}\label{sec:conclusion}

We exhibited problems with the EPR argument,
when seen through the eyes of mathematical logic, without
any reference to Bell's inequality.
We~also questioned the narrow-minded view of ``states'' taken in
that paper, although that view is not surprising from advocates
of ``elements of reality''.
Speaking of which, we should like to add to
the criticisms laboured in our paper that the notion
of ``elements of reality'' is left rather vague in the EPR paper.
Nevertheless, we came down even harder on Bohr's response which,
in our opinion, had entirely failed to address the main issue raised by~EPR.

In retrospect, our criticism of the EPR paper notwithstanding,
we admit without shame that we would have been more likely to side with them
than with Bohr\ldots\ until the bolt gave way to thunder under Bell's pen!

\section*{Acknowledgements}

The authors are grateful to Jeffrey Bub, Hilary Carteret, Christopher Fuchs
and N.~David Mermin for their comments and for helping us interpret
both the EPR paper and Bohr's response.
G.\,B. is supported in part by the Natural
Sciences and Engineering Research Council of Canada,
the Canada Research Chair programme and
the Canadian Institute for Advanced Research.

redpajama_set_name: RedPajamaArXiv