text:
\section{Introduction}

Using computer vision for navigation has long been well established, as a camera sensor is very easy to set up, cheap and power efficient. The main uses are for odometry and
3D
maps which are then used to control the navigation,
especially find a path and avoid obstacles.

Estimating depth from a camera is not an easy task, and validation data is very hard to obtain.
Indeed, knowing depth requires to know the 3-dimensional environment the camera is facing with respect to its position. This requires to
explicitly measure depth
with
range
sensors like Lidar or ToF cameras.

A major example of depth validation dataset is KITTI \cite{geiger2012we}, where a set of cameras and a Lidar are mounted on
a car. Following the acquisition,
the Lidar and video signals are calibrated and synchronized in order to construct sparse depth maps for every camera at every moment. The main problem with this method is that you need to construct a rigid rig between a Lidar and a camera, which, in addition to
being
very costly can become very heavy, and
is
not suitable to recreate the natural movement of a handheld camera or a consumer UAV camera.

To
address
this problem, we propose a way to construct a depth dataset with a two-step method that first uses a Lidar to scan an environment and then localizes images from a video camera with respect to the Lidar point cloud. Our method aims at being the most user friendly possible,
and with maximal flexibility, both
on the methods to construct the point cloud, and
on the type of camera used for acquisition.

Along with this paper, we provide the tool as an open source package thoroughly tested with an industrial research team to ensure its usability. We will present the general method that is widely inspired from ETH3D \cite{schoeps2017cvpr}, and present three use cases that we were able to construct. Finally, we will use these datasets to show with a benchmark on
monocular depth estimation
algorithms that results can vary greatly depending on the context and that the in-car environment, well referenced
through the KITTI benchmarks,
is far from being representative of all navigation use cases. This means that for
each new
navigation scenario, a new dataset should be constructed, at least for validation, which is exactly what our tool
aims at making
easier and cheaper.

\section{Related works}
\subsection{Depth estimation for navigation}
Depth estimation, and more generally 3D perception is a core task for autonomous navigation.
In the context of very light vehicles such as UAV, that can't carry heavy hardware, depth is often deduced from one or multiple cameras. The stereo camera is often used to compute depth from disparity \cite{nalpantidis2009stereovision},
that can also be estimated
with a single camera, which has the advantage of being much cheaper to integrate. Depth can then be deduced from motion, by using methods based on epipolar geometry \cite{hartley_zisserman_2004}. Structure from motion and SLAM techniques can then be used to deduce both depth and
camera
movement
using only
optical flow and geometric equations. It must be noted that all structure from motion algorithms
require rigid scenes,
i.e.
without moving objects.

More recently,
depth inference networks have been shown to be able to estimate depth from
a single
image solely from perspective and context. Indeed, with only one image, they were able to get reasonable scale invariant
(i.e. relative depth maps)
quality measures \cite{FuCVPR18-DORN,lee2019big,saxena20083,eigen2014depth,zhou2017unsupervised}.
However, in a navigation context, the scale invariant quality is not really interesting without a way to link the estimation to the real world.
Instead
of relative or normalized
values,
absolute depths are needed
for a navigation algorithm such as Model Predictive Control \cite{lopez2017aggressive} to be used.
To this end,
depth algorithms need to estimate both depth and movement in order to work for navigation.
Among the examples mentioned above,
Zhou \etal \cite{zhou2017unsupervised} is the only one we can work with.
It is important to note
that
depth from a single image does not use pixel displacements. It is deemed a more bio-inspired method which makes use of end-to-end training of convolutional neural networks and their capacity of generalizing implicit pixel structure, but it does not rely on
elaborate geometric constraints.
As such, it is
expected
to
be less robust for unusual scene, especially with a confusing perspective.

\subsection{Validation sets and benchmarks: the specific case of consumer UAV cameras, or why we need a new dataset}
This project was mostly motivated by the specific use case of depth estimation from UAV consumer drone, which lacks a proper validation dataset. Indeed, most depth algorithms are currently tested either for autonomous driving environments such as KITTI \cite{Uhrig2017THREEDV}, or indoor environment such as NYUv2 \cite{Silberman:ECCV12}. In these evaluation frameworks, single image depth prediction algorithms heavily prevail
within
the leaderboard, which makes these techniques {\em de facto} state of the art.

As shown
in
\cite{pinard2018learning}, the context of UAV navigation is much more heterogeneous than
these environments, both in terms of
camera pose
and surrounding environment. However, contrary to KITTI, the problem of moving objects is deemed (at least for now) a secondary problem when flying high above the ground. It is thus not certain that algorithms performing well on KITTI or NYUv2 will not perform poorly in this context. This is especially true when considering that moving objects in KITTI make depth from motion much harder, while depth from context is not robust enough to the viewpoint variability. As such, these datasets address issues for particular use cases with their own difficulties that are not reflected in the UAV use case and vice-versa. This idea is corroborated by the fact that the Sintel depth dataset \cite{Butler:ECCV:2012} remains largely unsolved for the moment because in addition to moving objects, scene heterogeneity is much more prevalent.

Inspired by
the
Flying Chairs
synthetic dataset
for optical flow\cite{DFIB15}, which
is founded on a surrealistic abstraction of the difficulties of optical flow,
Still Box \cite{isprs-annals-IV-2-W3-67-2017} has been proposed. It is a synthetic dataset trying to recreate the difficulties of a UAV flight, focusing on heterogeneity
of appearance,
using
random shapes and textures. As such, it is a good training dataset, the same way Flying chairs is a good training dataset for optical flow, but
obviously,
it is not suited for evaluation.
Table \ref{tab:datasets}, which compares the difficulties of the datasets currently used as benchmarks for depth quality, shows that none of them
is relevant for the UAV camera use case.

\begin{table*}
    \resizebox{\textwidth}{!}{
    \begin{tabular}{c|c|c|c|c|c}
         & KITTI\cite{Uhrig2017THREEDV} & NYUv2\cite{Silberman:ECCV12} & Sintel Depth\cite{Butler:ECCV:2012} & Still Box \cite{isprs-annals-IV-2-W3-67-2017} & a good UAV dataset \\
         \hline
        Moving objects & \cmark & \xmark & \cmark & \xmark & \xmark \\
        Outdoor & \cmark & \xmark & \cmark & N/A & \cmark \\
        Camera orientation variation & \xmark & \cmark & \cmark & \cmark & \cmark \\
        Camera position variation & \xmark & \xmark & \cmark & \cmark & \cmark \\
        Real videos & \cmark & \cmark & \xmark & \xmark & \cmark
    \end{tabular}
    }
    \caption{Difficulties featured in

    existing datasets.
    Being non photo-realistic, the distinction between Indoor and Outdoor is irrelevant for Still Box.}
    \label{tab:datasets}
\end{table*}

\subsection{Constructing a depth enabled dataset}
The
basic
principle of
depth enabled dataset construction
is to use a device
with reliable depth estimation capability,
that won't be available during evaluation.
Typically, we can use a rig with a RGB camera and a depth sensor like structured light \cite{Silberman:ECCV12}, Time of Flight, embedded Lidar \cite{Uhrig2017THREEDV, diode_dataset}, or light-field camera grids \cite{Schilling_MG-BDDPBL}. For evaluation, only the camera will be available, and the evaluation step will then measure the agreement between "reliable depth" measured by the dedicated sensor and estimated depth. It is important to note that an evaluation is only informative up to a certain point, where the quality of both methods are comparable. For dedicated depth sensors, we usually rely on the vendor's datasheet. However, this solution requires that we can emulate the camera movement of the use case we are trying to cover, \ie without potentially heavy depth sensors. This is problematic for consumer UAV, because the typical movement of its camera is difficult to mimic. Not only the camera is well stabilized, with a very smooth trajectory, that is not reproducible by hand, but the size of these cameras make it easy to fly very close to obstacles, which is not reproducible by a heavier UAV that could carry an additional depth sensor. These kinds of UAVs are usually very dangerous and need to be operated far from obstacles.

An interesting method for rigid scenes has been presented with ETH3D \cite{schoeps2017cvpr} and {\em Tanks and Temples} \cite{Knapitsch2017} that usually serve to evaluate photogrammetry. Instead of having a depth sensor and a camera attached to the same rig, the data acquisition is done in two steps. They first get a point cloud measure from fixed laser scanners such as the FARO Focus, and then take images from cameras in the same scene after removing the laser scanner. This implies that no object is moving in the scene, but allows for any camera to be used. In the particular case of ETH3D, they added a photogrammetry step, using the taken pictures and some RGBD rendering of the Lidar scanner with the COLMAP software \cite{schoenberger2016sfm}. This allowed them to get frame localization with respect to each other, but also with respect to the Lidar scanner position. With this technique, they were able to get depth map indirectly
from
the Lidar point cloud. They then used the generated depth map and camera position to get a stereo validation, with rectified frames. The ETH3D dataset construction method has been used as a foundation
of our framework.
Our goal is to generalize their method not only for photogrammetry oriented footage, but for all kinds of videos, using anything available to acquire a
3D
point cloud of a scene.

\section{Dataset creation method}

\subsection{The foundations: COLMAP and ETH3D}
ETH3D already offers to compute depth maps from a Lidar point cloud and a
separate
camera, but it has been used in a very particular context and some of its methods are not suited for a more general one. Because this article aims at using the same tools as ETH3D, we analyze them and indicate what aspects need to be changed in order to increase flexibility.

\subsubsection{Photogrammetry with COLMAP}
COLMAP \cite{schoenberger2016mvs, schoenberger2016sfm}  is a photogrammetry tool designed to be very robust, in order to reconstruct a 3D model from
crowdsourced
images taken
from
the internet. To go from a set of images to a 3D model with images localization, the steps applied are presented in Fig.~\ref{fig:colmap}.
The main problem of COLMAP is that even if the matching process can be dramatically accelerated with vocabulary tree matching \cite{schoenberger2016vote},
it is still in
$\mathcal{O}(n^3)$ where $n$ is the number of images. As such, the reconstruction process can be very long for videos.

\begin{figure*}
\tikzset{%
  >={Latex[width=2mm,length=2mm]},

            base/.style = {rectangle,draw=black,
                           minimum width=0.1cm, minimum height=1cm, align=center},
            io/.style = {base, fill=blue!30},
            op/.style = {base, fill=red!30},
            mapper/.style = {base, fill=green!30},
            kmeans/.style = {base, fill=orange!15},
            network/.style = {base, fill=purple!30}
}
\begin{tikzpicture}[font=\sffamily\small]
\begin{scope}[node distance=3cm and 1cm]

  \node (img)             [io,text width=2.2cm]              {Images};
  \node (fe)      [op, right of=img,text width=2.2cm]          {Feature \\ extraction (SIFT)};
  \node (fm)      [op, right of=fe,text width=2.2cm]   {Feature matching};
  \matrix (M) [matrix of nodes, mapper, right of=fm, xshift=1.5cm, yshift=-0.7cm, column sep=0.5cm, row sep=0.5cm, nodes={op, text width=2cm}]{
    Image registration & {Point \\ triangulation} \\
    Bundle Adjustment\\
  };
  \node (output)  [io, below of=M, text width=2.2cm, xshift=-1.5cm, yshift=0.5cm] {Output COLMAP model};
  \node (spm)      [io, below of=M,text width=2.2cm, xshift=1.5cm, yshift=0.5cm]   {Sparse 3D point cloud};
  \node (pms)      [op, left of=output,text width=2.2cm]   {PatchMatch stereo};
  \node (ecm)     [io, above of=M, text width=2.2cm, yshift=-0.5cm] {Already existing COLMAP model};
  \node (dpc)     [io, left of=pms, text width=2.2cm, yshift=0.6cm] {Dense point cloud};
  \node (vf)      [io, left of=pms, text width=2.2cm, yshift=-0.6cm] {Visibility index};
  \node (dm)      [op, left of=vf, text width=2.2cm, yshift=0.5cm] {Delaunay Mesher};
  \node (fmesh)      [io, left of=dm, text width=2.2cm] {Final mesh};

  \draw[->]     (img) -- (fe);
  \draw[->]     (fe) -- (fm);
  \draw[->]     (fm) -- (M-1-1);
  \draw[->]     (M-1-1) -- (M-1-2);
  \draw[->]     (M-1-2) |- (M-2-1);
  \draw[->]     (M-2-1) -- (M-1-1);
  \draw[->]     (M) -- (spm);
  \draw[->]     (output) -- (pms);
  \draw[->]     (ecm) -- (M);
  \draw[->]     (pms) -- (dpc);
  \draw[->]     (pms) -- (vf);
  \draw[->]     (dpc) -- (dm);
  \draw[->]     (vf) -- (dm);
  \draw[->]     (dm) -- (fmesh);
  \draw[->]     (M) -- (output);
  \draw[->]     (output) -- +(0,-1.3cm) -| (dm);
  \end{scope}
  \end{tikzpicture}
\linebreak
\caption{Photogrammetry workflow used in COLMAP. Note that the mapping process can stop at "Image registration" and the Delaunay mesher accepts any point cloud as long as the visibility index is correct.}
\label{fig:colmap}
\end{figure*}

\subsubsection{Depth generation with ETH3D}
ETH3D \cite{schoeps2017cvpr} uses COLMAP to localize calibrated images with respect to a Lidar point cloud taken with a FARO Focus. The FARO Focus is a fixed point Lidar that renders colored 3D points. Since it is fixed, every 3D point is measured from the same origin. This device allows them to synthesize high quality
depth-valued images with known position of the scanner (\ie the cloud origin)
that can be integrated in the COLMAP reconstruction process. The position of each image is thus known with respect to the point cloud. Each image then gets its position refined
by matching
feature points
between real images and rendered images from the colored point cloud
of an equivalent camera at the estimated position. However, the localisation part is not usable for our use case, because we want to be able to use
any
point cloud,
\ie
without colors. We can note that, as it is stated by their paper, the localization step can be done when simply constructing a 3D model with COLMAP and then register it with respect to the Lidar point cloud, with \eg ICP \cite{121791}.

Once the Images are localized with respect to Lidar, the depth rendering part is detailed in Figure~\ref{fig:eth3d}. Mainly, in addition to image calibration and localization, an occlusion mesh needs to be computed from the point cloud, which is then used to construct an occlusion depth. This depth map is
significantly
worse than the one from the point cloud, but it is only used to determine occlusion (and thus point visibility) and then avoid the ghosting effect of seeing through a 3D object
due to the sparsity of
the point cloud.
To this occlusion mesh we can add the "splats", which are created from isolated points: assuming
such
points are representative of thin objects (such as the leg of a chair), they may not be represented by the occlusion mesh but still count as occluding point, the splat creator is used to construct oriented squares at the point position, in order to avoid the risk of rendering
depth values of a background object for the rest of the thin object.

In ETH3D\cite{schoeps2017cvpr}, the occlusion mesh is constructed with Poisson algorithm \cite{kazhdan2006poisson}. This supposes that point normals can be computed and oriented. Again, this is possible
with point clouds
coming from a fixed laser scanner where normals are always oriented toward the origin, but it is not always the case for unstructured point clouds. We will
then
have to find an alternative solution.

\begin{figure*}\centering
\tikzset{%
  >={Latex[width=2mm,length=2mm]},

            base/.style = {rectangle,draw=black,
                           minimum width=0.1cm, minimum height=1cm, align=center},
            io/.style = {base, fill=blue!30},
            op/.style = {base, fill=red!30},
            mapper/.style = {base, fill=green!30},
            kmeans/.style = {base, fill=orange!15},
            network/.style = {base, fill=purple!30}
}

\begin{tikzpicture}[font=\sffamily\small]
\begin{scope}[node distance=3cm and 1cm]

  \node (cm)             [io,text width=2.2cm]              {COLMAP model};
  \node (img)   [io, right of=cm, text width=2.5cm] {localized and calibrated Image};
  \node (rlpc)  [io, below of=cm, text width=2.2cm, yshift=1.5cm] {Registered lidar point cloud};
  \node (oc)    [io, below of=rlpc,text width=2.2cm, yshift=1.5cm]          {Occlusion Mesh};
  \node (sc)    [op, right of=oc,text width=2.2cm, yshift=0.8cm]   {Splat Creator};
  \node (sp)    [io, below of=sc, text width=2.2cm, yshift=1.7cm] {Splats};
  \matrix (gtc) [matrix of nodes, mapper, right of=img, xshift=2.5cm, yshift=-2.5cm, column sep=0.5cm, row sep=0.5cm, nodes={op, text width=2.2cm}]{
    Depth from Lidar & Discard if

    beyond

    occlusion depth \\
    Occlusion Depth\\
  };
  \node (gtd)     [io, right of=gtc-1-2] {Ground truth \\ depth};
  \node (odo)     [io, right of=img, xshift=7cm] {Ground truth \\ odometry};

  \draw[->]     (cm) -- (img);
  \draw[->]     (img) -| (gtc-1-1);
  \draw[->]     (img) -- (gtc-2-1);
  \draw[->]     (img) -- (odo);
  \draw[->]     (rlpc) -- (gtc-1-1);
  \draw[->]     (rlpc) -- (sc);
  \draw[->]     (sp) -- (gtc-2-1);
  \draw[->]     (oc) |- +(5,-1.6) -| (gtc-2-1);
  \draw[->]     (gtc-1-1) -- (gtc-1-2);
  \draw[->]     (gtc-2-1) -- (gtc-1-2);

  \draw[->]     (oc) -- (sc);
  \draw[->]     (sc) -- (sp);
  \draw[->]     (gtc-1-2) -- (gtd);
  \end{scope}
  \end{tikzpicture}
\linebreak
\caption{Photogrammetry workflow used with ETH3D. Note that this workflow does not include localization, which is much more complicated but is not available to us,
since we assume that we don't necessarily
have a colored point cloud.}
\label{fig:eth3d}
\end{figure*}

\subsection{Changing the ETH3D workflow}
The constraints from the ETH3D
use case context can be summarized this way:
\begin{itemize}
    \item The number of images is very

    small. However if
    we want to use videos, we might have thousands of frames to localize.
    \item The Lidar

    is
    a very high quality color-enabled fixed scanner.

    However we want to be able to use any collection of
    unordered point clouds,
    for which the computation of the mesh is much harder.
\end{itemize}

In short, we need to find a way to localize images with COLMAP in linear time while keeping a good 3D reconstruction, and we need a way to construct a good mesh from
unordered point clouds.

\subsubsection{Extending COLMAP reconstruction}

The issue regarding the number of images can be solved by simply using a subset of images for reconstruction. The reconstruction needs to be good enough to be precisely registered with respect to the Lidar point cloud. For
a small
subset of frames, the full structure from motion mapping process can be applied even if it is very expensive. However, for remaining frames, the 3D point cloud will only be marginally better since
their views
are supposedly already covered by nearby frames. We can then afford to only register with respect to the already existing reconstruction. This process is much less expensive, as it does not need to
perform
a global bundle adjustment
at each
frame.

The issue is now to choose a good subset of frames. This can be solved during the data acquisition step. Indeed, we can take a set of pictures dedicated to photogrammetry following guidelines in \cite{Photogrammetry}. The goal is, for a fixed number of pictures to be used in the mapping process, to have pictures with the most uniformly distributed position and orientation view points. This can be done for example with a UAV orbiting around a particular object or flying along a grid above the scene: pictures are sampled at a regular pace to ensure a good parallax between images.

\subsubsection{Constructing the Occlusion mesh}

In the hypothesis we were successful in localizing every image with respect to the Lidar point cloud, we can use a specific tool to construct a mesh. After point cloud densification with multi view synthesis and depth maps fusion, COLMAP outputs a point cloud with normals and a visibility index indicating from which frame each 3D point is visible. These two features are used for mesh reconstruction. Indeed, as discussed earlier, Poisson reconstruction \cite{kazhdan2006poisson} can be used thanks
to
normals, and Delaunay meshing \cite{delauney} can be used with visibility index. If we make the unrealistic assumption that both dense reconstruction from COLMAP and Lidar point cloud are perfect, we can easily transfer these features from one cloud to another. Although COLMAP is known to have a low recall,
by
discarding many points in textureless areas, in our experiments, we found that transferring feature from the nearest neighbor of each Lidar point was sufficient.

\subsection{Our final workflow}

The final version of our workflow can be found in Figure~\ref{fig:workflow},
that represents the complete algorithm
needed for ground truth creation from ETH3D.
We now present it
in details.

\begin{figure*}\centering
\tikzset{%
  >={Latex[width=2mm,length=2mm]},

            base/.style = {rectangle,draw=black,
                           minimum width=0.1cm, minimum height=1cm, align=center},
            io/.style = {base, fill=blue!30},
            op/.style = {base, fill=red!30},
            mapper/.style = {base, fill=green!30},
            kmeans/.style = {base, fill=orange!15},
            network/.style = {base, fill=purple!30}
}

\begin{tikzpicture}[font=\sffamily\small]
\begin{scope}[node distance=3cm]

  \node (lpc)             [io,text width=2.5cm]              {Lidar Point Cloud};
  \node (img)      [io, below of=lpc,text width=2.5cm, yshift=0.9cm]          {Photogrammetry Images};
  \node (fm)      [io, below of=img,text width=2.2cm, yshift=1.5cm]          {Frames metadata (if any)};
  \node (ftl)      [io, below of=fm,text width=2.2cm, yshift=1.5cm]          {Frames to localize};
  \node (os)     [op, right of=fm, text width=2.2cm]   {Optimal Sample};
  \node (sfm)      [op, right of=img,text width=2.2cm]   {COLMAP (SFM)};
  \node (geo)      [op, right of=sfm,text width=2.2cm]   {Geo registration};
  \node (dense)   [op, right of=geo,text width=2.2cm]   {COLMAP (Densification)};
  \node (dpc)     [io, right of=dense, text width=2.2cm, yshift=0.6cm] {Dense point cloud};
  \node (vf)      [io, right of=dense, text width=2.2cm, yshift=-0.6cm] {Visibility index};
  \node (icp)    [op, above of=dpc, text width=2.2cm, yshift=-1.5cm] {Registration (e.g. ICP)};
  \node (fr)     [op, below of=geo, text width=2.2cm, yshift=1.5cm] {Frame registration};
  \node (rpc)    [io, right of=icp, text width=2.2cm] {Registered Lidar point cloud};
  \node (vft)    [op, right of=dpc, text width=2.2cm, yshift=-0.6cm] {Visibility transfer};
  \node (dm)   [op, below of=vft, text width=2.2cm, yshift=1.5cm] {Delaunay mesher};
  \node (om)   [io, below of=dm, text width=2.2cm, yshift=1.5cm] {Occlusion Mesh};
  \node (lf)   [io, right of=fr, text width=2.2cm]{Localized Frames};

  \draw[->] (lpc) -- (icp);
  \draw[->] (img) -- (sfm);
  \draw[->] (ftl) -- (os);
  \draw[->] (fm) -- (os);
  \draw[->] (os) -- (sfm);
  \draw[->] (sfm) -- (geo);
  \draw[->] (geo) -- (fr);
  \draw[->] (geo) -- (dense);
  \draw[->] (dense) -- (dpc);
  \draw[->] (dense) -- (vf);
  \draw[->] (dpc) -- (icp);
  \draw[->] (icp) -- (rpc);
  \draw[->] (rpc) -- (vft);
  \draw[->] (dpc) -- (vft);
  \draw[->] (vf) -- (vft);
  \draw[->] (vft) -- (dm);
  \draw[->] (dm) -- (om);
  \draw[->] (ftl) -| (fr);
  \draw[->] (fr) -- (lf);
\end{scope}
\end{tikzpicture}
\linebreak
\caption{Simplified representation of our workflow before using ETH3D tools. Video frames get registered with respect to the reconstruction point cloud, along with the Lidar point cloud. As such, we can use ETH3D with the COLMAP model, the Occlusion mesh and the registered Lidar point cloud.}
\label{fig:workflow}
\end{figure*}

\subsubsection{Data acquisition and ground truth point cloud creation}
In order for our protocol to be easy to follow, this part is very open and only asks for a point cloud of the scene,
acquired by any mean, and
not necessarily colored. The point cloud will be used as a perfect ground truth, which means that subsequently constructed depth maps
can
only be as precise as this point cloud. As suggested by \cite{schoeps2017cvpr}, a tripod fixed Lidar scanner can be used in order to have the maximum precision and density (precision at the millimeter level). However, other Lidar sensors can be used, such as mobile Lidars attached to a UAV, or a human handle. Although they lack
precision (which is now at the centimeter level), their ease of use can be leveraged to have a much more complete point cloud, especially in cluttered environments or unreachable places like a building's roof. As a last resort, when no Lidar is available, even the result of a photogrammetry can be used, in order to compare a real-time depth algorithm to a thorough reconstruction algorithm that
does not sacrifice quality for speed.
Throughout this section, the point cloud that serves as a ground truth is referred to as Lidar point cloud, but everything applies the same for any other ground truth point cloud creation method.

\subsubsection{First thorough photogrammetry}
This first step uses COLMAP to construct a photogrammetry comprising a point cloud and the
viewpoint
position of every image that was successfully localized. The reconstructed point cloud will then be localized with respect to the ground truth point cloud. Note that this step is different from ETH3D since we don't necessarily have a colored point cloud, and thus cannot synthesize images
from Lidar data.

As said above, our goal is to make model reconstruction as efficient as possible. If we
had to use
all the video frames we want to localize, the reconstruction would be comprehensive, but also extremely long. Instead, in addition to the "photogrammetry frames", we only include a subset of each wanted video to the photogrammetry process, so that the reconstruction is not too long
while including sufficient number of
different view points.
The sampling can be based on frame rate (e.g. only take one frame per second), but it is not ideal when the camera is not moving. In the case of frames with displacement metadata, we can sample a more efficient subset by using K-means \cite{Macqueen67somemethods} on a 6D point cloud composed of frames positions and orientation. Note that the importance of orientation in sampling can be parameterized by weighting the orientation coordinates in the 6D point cloud.

\subsubsection{Registration of ground truth point cloud with respect to the output of photogrammetry}

This step requires to find the optimal rigid transformation (rotation, translation and scale) between the reconstructed point cloud and the Lidar point cloud. Assuming COLMAP's reconstruction is good enough, a simple ICP \cite{121791} or
related algorithm
like point-to-plane ICP \cite{CHEN1992145} can be used to align the two point clouds. It is probably the most
sensitive
step of the process, and requires a human supervision.

Indeed, ICP is a somewhat unstable process that needs assessment and a good initialization. As such, a human needs to thoroughly check the point cloud alignment and manually estimate a first rough transformation, with \eg point pair picking. This can be done by \eg meshlab \footnote{\url{https://www.meshlab.net/}} or Cloudcompare \footnote{\url{http://www.cloudcompare.org/}}.

\subsubsection{Video localization}
This step only uses the image registration tool from COLMAP applied to the existing reconstruction. It does not contribute to the point cloud reconstruction, and only uses local bundle adjustment to localize the frame with its neighbors. As such, this operation is much faster than
the whole reconstruction process,
where
point triangulation and global bundle adjustment was used.

\subsubsection{Localization filtering}
The main interest of COLMAP compared to other SLAM method is that everything is global. This way every frame is not only localized with respect to its neighbors, but also with
respect to any
other frame
with which it shares a sufficient field of view.
As such, if each video frame contains features matched in enough well globally localized photogrammetry frames, the localization does not drift with respect to the reconstruction. However, since odometry is not perfect either, we can end up with a noisy trajectory that would not be possible
for
a real camera
with bounded
acceleration. This is especially true for consumer UAVs which usually focus on
video
smoothness
using
a stabilized guimbal, for aesthetic purposes.

To reflect this observation, on each localized video, we apply a Savitzky-Golay filter \cite{doi:10.1021/ac60214a047} to both trajectory and orientation. This filter not only smoothes the movement, but also helps detecting outlier that were badly localized. This way, we can discard the frames for which the distance between estimated
and filtered 6D positions
is above a certain threshold, and interpolate their position from neighboring frames. These frames will not be used for depth evaluation, but can be used \eg for depth algorithms that rely on odometry.

\subsubsection{Depth and pose ground truth generation}
Finally, we use the ground truth creator developed for ETH3D \cite{schoeps2017cvpr} to construct depth maps for all the video frames. We now have a ground truth for odometry and depth for each successfully localized frame.

\subsubsection{Dataset conversion and evaluation subset creation}

Now that we have images with odometry and depth, we can convert the dataset
format in compliance with more popular
datasets. For example, we can use the same format as KITTI odometry, in order to ease
validation
of depth algorithms on new datasets.

In addition, the same way Eigen\cite{eigen2014depth} proposed an evaluation split for depth, we can set a list of frames that will be used for depth evaluation. Odometry
can allow us to adapt our subset and its format to
several evaluation scenarios:
\begin{itemize}
    \item We can filter candidate frames

    according to the
    movement, \eg only forward motion (like in the context of a car), or without rotation.
    \item For algorithms based on normalized
    (relative)
    depth with corresponding pose estimation with respect to previous frames (\eg SFMLearner \cite{zhou2017unsupervised}), we can solve the scale factor with displacement magnitude, as suggested in \cite{pinard2018learning}.
    \item For algorithms that need odometry, such as multi view stereo, or algorithms that need frames with compensated

    rotation
    \cite{pinard2018learning}, it can be provided. This scenario is realistic for navigation context in the case of a UAV, where velocity and orientation need to be known primarily for a stable flight and a smooth video, and thus are available for these algorithms.
\end{itemize}

\subsection{Automation}
All the above processes have been included
into
a script that makes extensive use of COLMAP, ETH3D, PCL, and Parrot's Anafi SDK. This script is intended to be as easy to use and as flexible as possible, in order to cover
a wide range of
use cases and budgets.
It is open-sourced on Github with extensive usage documentation. \footnote{\url{https://github.com/ClementPinard/depth-dataset-builder/}}

\subsection{How good
a ground truth can we hope to reach?}

During the whole process, we have assumed that the 3D point cloud was perfect, or at least not improvable. We chose to trust precision ranges given by scanner vendors. For example for a fixed Lidar scanner such as the Faro Focus, the precision is below the centimeter, while for handheld Lidars such as Velodyne VLP16 used either with a UAV or handheld, the precision is below 5 centimeters. This means that depending on the device used for mapping, one needs to pay attention to the depth ranges seen during videos to
be localized.
This is particularly true for videos very close to obstacles, where the precision of depth maps measured by COLMAP's patch match stereo step can be better than Lidar reference.

Using fixed or mobile scanners is then a trade-off between mobility,
scan time,
precision and completeness.

Regarding odometry, we consider the localization made by COLMAP to be perfect. It can be noted that this was not the case for ETH3D, where they applied a pose fine tuning for each frame. Unfortunately, their method is only available for colored point clouds, which is not the case for most mobile Lidars. This problem can be mitigated by two factors:
\begin{itemize}
    \item The colorless scanners are also the

    less precise ones.
    As such, solving this problem might get negligible improvement since the point cloud quality will then

    become the limiting factor.
    Otherwise, we can simply apply

    ETH3D's
    pose refinement technique.
    \item We tested COLMAP on EuroC dataset, and were surprised to see that odometry from

    the provided ground truth
    (measured with a Lidar scanner and an IMU) was not very good compared to odometry computed by COLMAP. This can be seen on Figure \ref{fig:euroc_prob}, where the triangulated 3d points are visibly much less noisy from COLMAP odometry than from ground truth. This is corroborated with the EuroC depth dataset proposed by \cite{Gordon_2019_ICCV}, where synthesized depth maps from frame position and camera calibration were not exactly aligned with the camera, even in their illustrating figure (see Figure \ref{fig:wild}).
\end{itemize}
In sum, we have good, albeit subjective, reasons
to think that the odometry from dedicated sensors is not necessarily needed compared to the one computed by COLMAP. We can only make a subjective manual assessment, but as mentioned in the first point, we believe that point cloud quality is often the limiting factor. However, we cannot ignore the fact that for some particular cases, such as a video that is isolated in a cluttered part of a scene and only connected to the rest of the photogrammetry by a few frames, the odometry can drift. This makes these localized frames misaligned with the Lidar point cloud, and thus
with
a poor ground truth depth, even if the Lidar point cloud registration step is optimal, because it's only a rigid transformation. A solution to this problem could be to apply a non-rigid registration \cite{6361384} of the COLMAP point cloud, deforming the cloud and
then
also the frame localization to fit the Lidar point cloud more precisely.
This might be the occasion of a future work combining COLMAP's bundle adjustment and cloud-to-cloud distance between COLMAP and Lidar point cloud. However, a more direct way of limiting this problem is to ensure during data acquisition that all video frames can be localized with a large number of photogrammetry oriented pictures, so that a "loop closure" step is applied very often, with pictures designed to have a very precise localization with respect to the reconstruction cloud.

\begin{figure*}
    \centering

    \begin{tabular}{cc}
         \includegraphics[width=0.49\textwidth]{images/colmap_problem.png} &
         \includegraphics[width=0.49\textwidth]{images/colmap_no_problem.png}\\
         (a) & (b)\\
    \end{tabular}
    \caption{

    Visual qualitative assessment
    from COLMAP mapping process.

    (a)~localization

    from available ground truth odometry, measured from Lidar and IMU.

    (b)~localization

    deduced by COLMAP during the mapping process

    with
    SLAM. Visualization was done via COLMAP GUI.}
    \label{fig:euroc_prob}
\end{figure*}

\begin{figure}
    \centering
    \def0.5{0.5}
    \setlength{\tabcolsep}{1pt}
     \begin{tabular}{cc}
     \includegraphics[width=.59\columnwidth]{images/problem_depth.png} &
      \includegraphics[width=.39\columnwidth]{images/problem_depth_zoom.png}\\ (a) & (b)
      \end{tabular}
     \begin{tabular}{c}
     \includegraphics[width=\columnwidth]{images/problem_depth2.png}\\
     (c)
     \end{tabular}
    \caption{Depth map proposed by \cite{Gordon_2019_ICCV}. Image and colored depth maps are superposed to show the alignment problem, especially for foreground and background delimitation.
    (a)~depth
    map shown in Figure SM2 of \cite{Gordon_2019_ICCV} (from their supplementary materials), with
    (b)~zoom
    details.

    (c)~depth
    map from another sequence, where alignment error is larger than 40 pixels.}
    \label{fig:wild}
\end{figure}

\section{Measuring depth quality for navigation purposes}
Since we
want to build
an evaluation dataset, we are
looking for
the most informative metrics. As we are mostly interested in the context of navigation, we chose to follow the methodology proposed
in
\cite{pinard2018learning}. Namely, contrary to the well used "Eigen-split" \cite{eigen2014depth}, using metrics from Garg \etal \cite{garg2016unsupervised}, we want to apply a realistic navigation scenario,
that requires
the estimated depth
to be absolute, and not up to a factor computed with the ratio of medians. This makes depth from single view algorithms unable to compete, unless there is a way to compute the scale factor by comparing odometry with actual displacement, which is much more realistic for any kind of autonomous vehicle. Single frames algorithms such as Deep Ordinal Regression Network (DORN)~\cite{FuCVPR18-DORN} or Big to Small (bts)~\cite{lee2019big} can thus not be used, but some other algorithms such as SFMLearner and its
variants~
\cite{zhou2017unsupervised, monodepth2} are trained with a pose estimator and thus can be evaluated.

\subsection{On the information a metric gives}
Most of the time, for evaluation datasets, metrics are used for the only purpose of ranking different algorithms. This is useful for benchmarking and choosing an existing algorithm for one's usage, but it raises some problems from the end user's perspective:
\begin{itemize}
    \item It does not reflect the context of the use case that might be different from the original validation set. For example, if someone wants to estimate depth from a single camera but only for close objects because the long range is already covered by another sensor, this user won't be interested in long range depth estimation quality, and the metric used for benchmarking will have to reflect this.
    \item Some characteristics of an algorithm are inherently

    antagonistic,
    and thus a trade-off must be decided between

    them.
    The most usual example is accuracy {\em vs} speed. By ranking algorithms with only one metric, the dataset makes the trade-off decision in place of the end user and thus takes the risk of not being informative. For example, because speed is only given as declared from the authors, KITTI depth benchmark \cite{Uhrig2017THREEDV} completely disregards it. It would be interesting to have data presented into a 2D chart, the same way as VOT \cite{VOT_TPAMI} so that the end user can eventually

    set his own
    speed/accuracy trade-off.

    \item A metric can give information about the distribution of expected values given a particular estimation. As such, as shown

    in~
    \cite[p.~38]{pinardphd} all metrics are not the same when it comes to trying to characterize possible real values of an estimation. The ideal solution would be to give for each estimated depth value the exact distribution of real depth. That would require an infinite set, but the validation set can give an approximation that gives more insight than just a single number.

\end{itemize}

To reflect those
practices,
we decide to have both a set of classic metrics and histograms to give as much information as possible for a given algorithm. Again, the code for metric measurement is open sourced on Github \footnote{\url{https://github.com/ClementPinard/depth-dataset-builder/tree/master/evaluation_toolkit}}. It consists in getting all depth pixels and their estimation, within an unordered set $V$. Note that this evaluation set is not image-wise, all depth values are collected at first with corresponding metadata, and the metric computation is done at the end. This is useful for images with a very sparse depth, where no representative statistics can be computed.

\subsection{Scalar metrics}
A scalar metric is obtained by computing a depth
errors for all points and then averaging
them. As mentioned above, the mean is global
over the validation set $V$:
\begin{equation*}
\mathcal{E}_f = \frac{1}{|V|}\sum_{(\widetilde{\theta}, \theta)\in V} f(\widetilde{\theta}, \theta)
= \mathbb{E}_V(f(\widetilde{\theta}, \theta ))
\end{equation*}

Where $\theta$ is the ground truth depth for a particular pixel,
$\widetilde{\theta}$ its estimate,
and $f$ the error function.

Table \ref{tab:error_functions} shows provided error functions and corresponding names.

\begin{table*}
    \renewcommand{0.5}{1.5}
    \centering
    \begin{tabular}{|l|c|c|}
    \hline
    Error Name & Acronym &

    $\mathbb{E}(f(\widetilde{\theta}, \theta ))$ \\ \hline
     Mean Absolute Error & $MAE$ & $\mathbb{E}|\widetilde{\theta} - \theta |$ \\ \hline
     Mean Relative Error & $MRE$ & $\mathbb{E}\frac{|\widetilde{\theta} - \theta|}{\theta}$ \\ \hline
     Mean Log Error & $MLE$ & $ \mathbb{E}|\log(\widetilde{\theta}) - \log(\theta)|$
     \\ \hline
     Standard Absolute Error & $SAE$ & $\sqrt{\mathbb{E}(\widetilde{\theta} - \theta )^2}$ \\ \hline
     Standard Log Error & $SLE$ & $\sqrt{\mathbb{E}(\log(\widetilde{\theta}) - \log(\theta))^2}$ \\ \hline
     Precisions $\delta$ & $P_\delta$ & $P\left(\left|\log\left(\frac{\widetilde{\theta}}{\theta}\right)\right| \leq \log(\delta) \right)$ \\ \hline
    \end{tabular}
    \caption{Considered

    Metrics
    Summary}
    \label{tab:error_functions}
\end{table*}

\subsection{Histogram metrics}

Histogram metrics cannot be used for ranking algorithms but they give much more information. We propose two different histograms:
\begin{itemize}
    \item Depth wise error: Given a scalar metric, we can compute the error for particular depth values.

    $$

    H_D(\theta_0)
    = \mathbb{E}_{\{ \theta = \theta_0 \}}(f(\theta, \widetilde{\theta}))$$

    \item Difference distribution: This

    distribution

    is normally centered around $0$, and its standard deviation is the standard error mentioned

    on Table~\ref{tab:error_functions}.
    Having the whole distribution is especially interesting for distribution that are not symmetrical. A safety interval can then be deduced from this histogram for each side of estimated depth. For this part, we follow the assumptions that the log of depth estimation is more likely to follow a symmetrical distribution than raw depth estimation. It is equivalent to get a distribution of the ratio between

    estimation and ground truth,
    usually centered around 1.

    $$

    H_{\Delta}(\delta) =  P(\log(\widetilde{\theta}) - \log(\theta) = \delta) = P\left(\frac{\widetilde{\theta}}{\theta} = 10^\delta\right) $$

\end{itemize}

\subsection{Displacement wise metrics}
In addition to those metrics, we propose to have a displacement wise metric in the form of an histogram. Namely, if we know the displacement, we can deduce what point in the image the camera is moving toward to.
This point, called the flight path vector (FPV) for aircraft, also corresponds to the epipole for multi-view geometry, and to the focus of expansion (FOE) for optical flow in the case of rotation-less movement. As a consequence, these points are deemed more important than the other ones.

$$\mathcal{E}_{FPV}(\alpha) = \mathbb{E}_{V(\alpha)}(f(\widetilde{\theta}, \theta ))$$

Where $V(\alpha)$ is the set of points at a distance $\alpha$ from the FPV of each image (be it in pixels, or in radians).

This particular distribution can help discard an algorithm that fails to estimate
depth around
those points while having good metrics otherwise. This is the case for optical flow based method for a stabilized camera without rotation: optical flow is too
small
around the FPV and thus
disparity based depth estimation
becomes too noisy. \cite[p~32]{pinardphd}

\section{Applications}

\subsection{Preamble}

In this section, we present two use cases we have covered using this tool. They both feature drone footage, and
are used to test
two algorithms: DepthNet \cite{isprs-annals-IV-2-W3-67-2017} and SFMLearner \cite{zhou2017unsupervised}.
They
are not state of the art on
popular
benchmarks like KITTI\cite{Uhrig2017THREEDV},
but they are relevant to demonstrate the interest of the proposed evaluation framework and data sets, for the following reasons:
\begin{itemize}
    \item They are both compatible with our evaluation scenario focused on navigation, because we can scale their depth estimation only using odometry
    \item They are supposed to be real time, contrary to MVS algorithms like the one used in COLMAP
    \item
    Their performance metrics are similar
    on KITTI depth \cite{Uhrig2017THREEDV}
    \item Contrary to SFMLearner, DepthNet has been specifically designed to be robust
    to variability of context and scene layout.
    As such, for heterogeneous data sets, we should expect DepthNet to perform better than SFMLearner
    \cite{schoenberger2016mvs}.
\end{itemize}

The evaluation protocol is the same for the two scenarios. We
divide the data set into two parts, one for training and one for test. Images and depth maps are rescaled to $416 \times 234$, and for both neural networks, we follow their publication training schedule. DepthNet is pre-trained on the Still Box dataset \cite{isprs-annals-IV-2-W3-67-2017},
then trained on the learning data set with photometric depth auto-supervision,
and
supervision for odometry rotation.
SFMLearner is not pre-trained (we did not see improvements when pre-training with KITTI);
it is
entirely
trained
on the learning data set
without any supervision.

\subsection{First application: The drone Manoir dataset}
\subsubsection{Context}
Our first
use case
is a scene with a mansion
(in French: manoir)
in the countryside, on a $350 \times 100 m^2$ terrain.
The maximum altitude of obstacle is $20 m$. 3D Lidar data was captured by a DJI Matrice 600 with a Velodyne VLP-16 on board, with RTK GPS system (see figure~\ref{fig:manoir_map}). The flight altitude of this UAV was 30 meters at minimum for safety reasons. The UAV was used because it can cover an area much faster than any fixed scanner, and can easily scan building roofs. The whole scanning process took less than 10 minutes, meaning we could have covered a very large area in less than a day.

For photogrammetry oriented pictures, we used an Anafi drone with the free Pix4D app that lets us make one grid and two orbits above the field we wanted to scan. We also used a personal DSLR (Sony alpha-6000) for additional photo. We took videos at two different quality settings for a total of 65k frames to localize:
 \begin{itemize}

     \item $3840 \times 2160$ (4K) at 30 fps, best quality setting.
     \item $1280 \times 720$ at 120 fps, bad quality but high frame rate.
 \end{itemize}
 See Figure~\ref{fig:manoir_photo}.

 \begin{figure}
     \centering
     \def0.5{0.5}
    \setlength{\tabcolsep}{1pt}
     \begin{tabular}{cc}
          \includegraphics[width=.49\columnwidth]{images/plan1.jpg} & \includegraphics[width=.49\columnwidth]{images/drone1.jpg} \\
          \includegraphics[width=.49\columnwidth]{images/pointcloud1.jpg} &
          \includegraphics[width=.49\columnwidth]{images/pointcloud2.jpg}
     \end{tabular}
     \caption{3D Mapping process of the Manoir dataset.
     Top:
     map of the

     scanned area
     and
     scan device used for mapping.

     Bottom: two views of the
     resulting point cloud.}
     \label{fig:manoir_map}
 \end{figure}

 \begin{figure}
     \centering
     \def0.5{0.5}
    \setlength{\tabcolsep}{1pt}
     \begin{tabular}{cc}
          \includegraphics[width=.49\columnwidth]{images/plan2.jpg} & \includegraphics[width=.45\columnwidth]{images/plan3.jpg} \\
          \includegraphics[width=.49\columnwidth]{images/piloting1.jpg} &
          \includegraphics[width=.49\columnwidth]{images/piloting2.jpg}
     \end{tabular}
     \caption{Video acquisition process of the Manoir dataset. Top: grid and orbit flight plan used for photogrammetry. Bottom: Video samples from the Anafi drone, 4k (left) and 720p (right)}
     \label{fig:manoir_photo}
 \end{figure}

\subsubsection{Result}
A
subjective result of photogrammetry
can be seen on
Figure~\ref{fig:manoir_photo_result}, for the optimal subset of 1000 frames and a full video taken by a drone. Finally, Figure~\ref{fig:manoir_full} shows a sample of computed depth maps. A video playlist is available as a supplementary material for all the sequences \footnote{\url{https://youtube.com/playlist?list=PLMeM2q87QjqhYA_LfJY925ZAGyD5cS6Q-}}

\begin{figure}
    \centering
    \includegraphics[width=.49\columnwidth]{images/photog1.jpg}
    \includegraphics[width=.49\columnwidth]{images/photog2.jpg}
    \caption{Left: global photogrammetry result
    (point cloud in black, camera odometry in red).
    Right:

    camera odometry during a given video footage (red curve), w.r.t.
    the global photogrammetry.}
    \label{fig:manoir_photo_result}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{images/manoir_result.jpg}
    \caption{Depth result from the Manoir dataset. From left to right, top to bottom: RGB image, depth map, RGB + depth, occlusion depth.}
    \label{fig:manoir_full}
\end{figure}

\subsubsection{Note on point cloud completeness}
As it can be seen in Figure~\ref{fig:manoir_map}, the point cloud seriously lacks completeness for parts that are not easily seen from an altitude of 30 meters. This includes vertical sections like walls or poles, like seen in Figure~\ref{fig:pcprob}, or cluttered sections, like under trees or
inside
a tunnel. From this
remark, we can do two observations:
\begin{itemize}
    \item Although it covers a

    large
    area very quickly, UAV Lidar scanning does not cover a wide range of viewpoints. The 30 meters altitude makes it difficult to see the same thing as within a few meters of altitude. As such, it would have been useful to add Lidar scans from a lower altitude, from other methods. For example, we can use fixed Lidar and handheld Lidar. Regarding this use case, we recommend to use a handheld Lidar because it's much faster than fixed Lidar

    (typically as fast as human gait,
    while fixed Lidar requires several minutes per viewpoint), and if a UAV is available, all the hardware for using Lidar with IMU technique like \eg LIO-SAM \cite{liosam2020shan} or a proprietary technique like GeoSLAM is already available.
    \item In case no other Lidar scan is possible (\eg because the area is not reachable other than with a drone), the COLMAP cloud is locally good. An interesting compromise for future work would be to combine the Lidar cloud with COLMAP reconstruction for obstacles very close to the camera.
\end{itemize}

\begin{figure}
    \centering
    \def0.5{0.5}
    \setlength{\tabcolsep}{1pt}
    \begin{tabular}{cc}
        \includegraphics[width=.33\columnwidth]{images/problem2.png} & \includegraphics[width=.66\columnwidth]{images/problem4.png} \\
        \includegraphics[width=.33\columnwidth]{images/problem1.png} &
        \includegraphics[width=.66\columnwidth]{images/problem3.png}
    \end{tabular}
    \caption{Images and corresponding occlusion depth maps illustrating problems
    coming
    from point cloud sparsity.}
    \label{fig:pcprob}
\end{figure}

\subsubsection{Benchmark}
Scalar metrics are shown in Table \ref{tab:manoir_metrics}.
Distributions of (estimation / ground truth) ratios (equivalent to
log depth difference, top),
and log-errors according to the estimated depth (bottom)
are shown on Figure~\ref{fig:hists} (left panel), while samples can be seen Figure~\ref{fig:samples} (first three rows). This first evaluation shows that unsurprisingly, DepthNet largely outperforms SFMLearner in all metrics. This is also visible on the log difference distribution: although both distributions have their maximum at a null log difference (
i.e.
a ratio of 1), indicating an unbiased estimator, DepthNet's distribution is much more concentrated. We can also see that both distribution exhibits a clear skewness toward negative difference. This is corroborated with the second plot where we can see that for both algorithms, the log error is much lower above 20 meters. This is a good hint for a potential drone manufacturer to think of a dedicated system for low depth values. For example, a stereo camera setup would be complementary to these two systems, because it's more accurate for lower depth values.

For conciseness purpose, we have not shown all the histograms our tool can generate. However, we have made them easily generated within the tool we open sourced. \footnote{\url{https://github.com/ClementPinard/depth-dataset-builder\#depth-algorithm-evaluation}}
\begin{table*}
    \centering
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|}
\hline
Method & \cellcolor{blue!15}MAE & \cellcolor{blue!15}MRE & \cellcolor{blue!15}MLE & \cellcolor{blue!15}SAE & \cellcolor{blue!15}SLE & \cellcolor{red!25}$P_{1.25}$ & \cellcolor{red!25}$P_{1.25^2}$ & \cellcolor{red!25}$P_{1.25^3}$ \\
\hline
SFMLearner \cite{zhou2017unsupervised} & 18.40 & 0.5145 & 1.005 & 24.46 & 1.458 & 0.2395 & 0.4113 & 0.5385 \\
DepthNet \cite{pinard2018learning} & \textbf{11.99} & \textbf{0.3275} & \textbf{0.5290} & \textbf{18.51} & \textbf{0.9099} & \textbf{0.5241} & \textbf{0.7069} & \textbf{0.7717} \\
\hline
\end{tabular}
    \caption{Metric comparison

    between
    SFMLearner \cite{zhou2017unsupervised} and DepthNet \cite{pinard2018learning} on {\em Manoir} dataset.}
    \label{tab:manoir_metrics}
\end{table*}

\subsection{Second application: University hall dataset}
\subsubsection{Context}
Our second
use case
is an indoor scene,
shown on
Figure~\ref{fig:ensta}, the hall of a University with a very high ceiling. For scan mapping, we use a handheld Zeb Horizon from GeoSLAM, with an announced precision of 4 cm. For videos, we used an Anafi; we have no photogrammetry-oriented frames.
Like
for Manoir dataset,
the results of our
small benchmark
can be seen on
Table~\ref{tab:ensta_metrics} and Figure~\ref{fig:hists} (right panel). Some samples can be seen Figure~\ref{fig:samples} (last two rows)

Although DepthNet still outperforms SFMLearner for most metrics, the estimation accuracy is much lower. This can be explained by the fact that the university hall is mostly composed of completely white walls, with many glass windows and a somewhat reflective ground. This is a very challenging use case for SFM based training and inference methods. These results indicate that both SFMLearner and DepthNet are much more suited for outdoor scenarios, with irregular textured surfaces, and that
active sensing solutions like structured light or time-of-flight are
probably more interesting for this dataset.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{images/ensta_hall.jpg}
    \caption{University hall depth visualization, same structure as Figure~\ref{fig:manoir_full}}
    \label{fig:ensta}
\end{figure}
\begin{table*}
    \centering
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|}
\hline
Method & \cellcolor{blue!15}MAE & \cellcolor{blue!15}MRE & \cellcolor{blue!15}MLE & \cellcolor{blue!15}SAE & \cellcolor{blue!15}SLE & \cellcolor{red!25}$P_{1.25}$ & \cellcolor{red!25}$P_{1.25^2}$ & \cellcolor{red!25}$P_{1.25^3}$ \\
\hline
SFMLearner \cite{zhou2017unsupervised} & 10.25 & 1.0225 & 0.5682 & 18.29 & 0.7798 & 0.3180 & 0.5461 & 0.6912\\
DepthNet \cite{pinard2018learning} & \textbf{5.890} & \textbf{0.5197} & \textbf{0.5056} & \textbf{10.612} & \textbf{0.726} & \textbf{0.3184} & \textbf{0.5982} & \textbf{0.7694} \\
\hline
\end{tabular}
    \caption{Metric comparison

    between
    SFMLearner \cite{zhou2017unsupervised} and DepthNet \cite{pinard2018learning} on {\em University hall} dataset}
    \label{tab:ensta_metrics}
\end{table*}

\begin{figure}
    \centering

    \resizebox{.45\columnwidth}{!}{
    \input{images/plots/my_graph_manoir.pgf}
    }
    \resizebox{.45\columnwidth}{!}{
    \input{images/plots/my_graph_ensta.pgf}
    }
    \caption{Comparison of error distributions between SFMLearner \cite{zhou2017unsupervised} and DepthNet \cite{pinard2018learning} on {\em Manoir} dataset (left) and on {\em University hall} dataset (right)
    Top:
    Distribution of ratios between estimation and ground truth (above 1 means estimation is higher), at log scale. This is equivalent to the log difference distribution.
    Bottom:

    Log error as a function of estimated depth.
    For visualization purpose, the $y$ scale is logarithmic. The

    curve
    shows the median value and the shaded area represents the $ 50\% $ confidence interval.
    }
    \label{fig:hists}
\end{figure}

\begin{figure*}
    \centering
    \begin{tabular}{@{\hskip 0.1em}c@{\hskip 0.1em}c@{\hskip 0.1em}c@{\hskip 0.1em}c}
    Image & GroudTruth & DepthNet\cite{isprs-annals-IV-2-W3-67-2017} & SFMLearner \cite{zhou2017unsupervised} \\
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_12462.jpg} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_12462_GT.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_12462_DepthNet.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_12462_SFMLearner.png} \\
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_13951.jpg} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_13951_GT.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_13951_DepthNet.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_13951_SFMLearner.png} \\
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_20162.jpg} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_20162_GT.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_20162_DepthNet.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P1890430_20162_SFMLearner.png} \\
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01535.jpg} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01535_GT.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01535_DepthNet.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01535_SFMLearner.png} \\
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01578.jpg} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01578_GT.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01578_DepthNet.png} &
    \includegraphics[width=.25\textwidth]{images/viz/P0230025_01578_SFMLearner.png}
    \end{tabular}
    \caption{Vizualisation of some samples on {\em Manoir} and {\em University hall} data sets. Colormap is
    OpenCV
    Rainbow, normalized to Ground truth.}
    \label{fig:samples}
\end{figure*}

\section{Discussion
and conclusive remarks}
In this paper, we presented
RDC,
a powerful tool for depth dataset generation. It is very flexible
with respect
to available budget and hardware. Its context of application can be as
rudimentary as no-budget-at-all,
only using a handheld camera (\eg the user's phone camera), and
still
it lets a user make the
most of
limited means, to have depth enabled videos with potentially infinite range. By
leaving open access to future users,
we hope to improve it by
gradually increasing the diversity of use cases.

\subsection{Limitations}
RDC still suffers from some
already known limitations:
\begin{itemize}
    \item The first limitation is obviously the

    need for a rigid
    scene. Although this problem can be mitigated in the case of a UAV flying, it can be particularly problematic for in-car environment in a urban area where many people and other vehicles are dynamic. It can also be a problem when Lidar scanning and video recording are not done at the same time, in a scene with rigid
    but movable
    objects.

    This was the case in the University hall dataset, where some seats had been moved, thus

    making some
    depth maps useless. A possible solution would be to combine a dense high quality point cloud of a scene, scanned beforehand, and sparse point cloud from a mobile Lidar.  By characterizing possible dynamic elements, like humans and other vehicles in a urban area, we may be able to discard the depth map from rigid scene where the dynamic

    elements are segmented,
    and replace it with the depth from the sparse mobile Lidar.
    \item Although it does not require heavy hardware for data acquisition, the computer used by this tool must be powerful (as

    with any
    photogrammetry task), and have plenty of disk space, which mitigates the fact that we adapt to any kind of budget. However, a regular gaming desktop computer is often enough, and is orders of magnitude cheaper than a full Lidar and cam rig solution.
    \item Although it can be improved by using better registration algorithms than a simple ICP, the registration part will often need a human

    eye
    to assess the registration quality. This part of the process lies in the middle of it, which makes the tool not totally automatic. It is essential that it happens after the photogrammetry and before ground truth generation, so there is no obvious solution yet.
    \item When the camera is located in a very sparse portion of the point cloud, the occlusion depth will be of poor quality. It is possible that the end user will discover after the whole dataset creation process that some scenes

    would
    need a better scanning process. It would

    then
    be interesting to

    enter
    a warning before the end of the process, as soon as the Lidar point cloud is registered, that some areas lack data.
\end{itemize}

\subsection{Further work}
The next logical step is now to construct a
more comprehensive
dataset with this tool, using different scenes, contexts, and modalities, in the same way ETH3D dataset was
built, but
with much more frames.

Additionally, a full evaluation suite would be very beneficial, as this
initial version
only provides
basic examples of histogram based algorithm quality assessment.

Finally, now that we have a way of localizing video frames in a point cloud, we can make
more ground truths to test
other kinds of algorithms:
\begin{itemize}
    \item From odometry and ground truth point cloud, ground truth

    optical flow can be deduced.
    \item If we annotate the Lidar point cloud with semantic parts, we can construct semantic segmentation maps.
\end{itemize}

\section*{Acknowledgements}
Acquisitions for the {\em Manoir} dataset were made in collaboration with AIRD'ECO-Drone\footnote{\url{https://www.airdeco-drone.com}} company, thanks to the financial support of Parrot\footnote{\url{https://www.parrot.com}} company.
Acquisitions for the {\em University hall} dataset were made entirely by Cl\'ement Pinard, thanks to the equipment and training provided by Geomesure\footnote{\url{https://www.geomesure.fr/}} company.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.
Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 4321 in the example.  To do this fine the line (around
line 23)
\begin{verbatim}
\setcounter{page}{4321}
\end{verbatim}
where the number 4321 is your assigned starting page.

Make sure the first page is numbered by commenting out the first page being
empty on line 46
\begin{verbatim}
\end{verbatim}

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv


See preview function bold font and download this cool font for free. The font is classified as bold fonts. This font viewed 7580 times and downloaded 2029 times. You can also downloads other fonts: Function Bold, Function LH Regular, Function Two Light Regular, Function Regular, Function Display Regular, Function Cond Bold for Windows and Mac at FreakFonts.com. If you like this font don't forget share them with you friends!


Discussion in 'MacBook Pro' started by beqbutu, Apr 14, 2014.
Hi guys i'm new here, recently i bought a macbook pro a1278 that had liquid damage, took it apart cleaned and put it back together and it worked but the dvd didn't, no signs of life at all and not recognised in os system info, i tried to put another one in but don't know if it works or not but still resulted with the same symptoms, should i try to get a new one? To test if any of the drives work i bought a usb enclosure today from ebay, has anyone had this sort of fault with the dvd drive as I'm thinking that its the motherboard thats gone bad, or my luck both dvd's have the same fault, the funny thing is that every other thing works perfectly well except the mic but its a minor thing and i can live without it.
It is VERY common for those Superdrives to fail.
What do you mean? Are you saying the new external USB SuperDrives fail?
I was referring to the internal Superdrive like the OP mentioned. I have not heard of any issues with the new external models.
Good. So would you recommend it? I plan to use it for films, X-Plane and backing up to DVD's.
I have one and it works well with my Macbook Air. Never had any trouble with it at all.
I know there are cheaper, generic drives that will work, but I have no experience with them.


Property Overview - Spacious & well-maintained two-story four-bedroom 2.5 bath home w/ a gorgeous Mount Rainier view! Located on a dead-end quiet street, close to shopping restaurants schools entertainment & JBLM. Bamboo flooring ceramic tile French doors great room floor plan w/large kitchen breakfast bar & computer nook. Formal living room. Master bedroom has 5 pc suite w/ luxurious soaking tub. Fenced backyard w/patio for barbecues & entertaining. Storage shed. Don?t miss the virtual tour! You?ll love it!
This property was not sold in the last 13 years.
Relisted $114,900 above previous listing price.
Light & Bright w/lots of space!4b/2.5bth w/fully fenced yard & 2car garage!TONS of gorgeous rose bushes & an apple tree!Cvd back deck. Vaulted ceilings give an open, bright atmosphere!Lovely gas fireplace & built in TV alcove in lvg rm!Range, microwave, and dishwasher stay in fabulous kit with maple cabinets & a built-in desk wkspace.HUGE walk-in closets throughout. Mstr boasts 5pc bath,w/soaking tub!Jack & Jill Bth connecting 2 upstairs rms!4th bdrm could also be den/study.
16566 Rainier View Dr SE is located in the city of Yelm, WA.
Are you hunting for property records for the home located at 16566 Rainier View Dr Se, Yelm, WA 98597? If so, we can help you easily find crucial details for the property as well as other Yelm property records, Washington public records, and other vital real estate details that can help you simplify the moving process. At this moment the home at 16566 Rainier View Dr SE is not currently for sale but we have other equally lovely homes in Yelm listed on realtor.com®. When you go through our all-inclusive database of Washington homes for sale and you will easily find the Yelm home for sale that is made just for you.


text:
Third Black Employee Joins Racial Discrimination Lawsuit Against Fox News
Longtime staffer Judy Slater was recently fired for alleged racist behavior
Brian Flood | April 4, 2017 @ 7:14 AM
A third black female employee in Fox News' payroll department has joined a racial discrimination lawsuit filed last week by two of her black colleagues, according to legal documents obtained by TheWrap.
The amended lawsuit, filed against the network and recently axed exec Judy Slater on Tuesday in New York State Supreme Court, alleges a broad pattern of top-down racial harassment that they claim the company tolerated for years.
Among the allegations: Slater told the third employee, credit collections manager Monica Douglas, that she had "black eyes" as opposed to the "Aryan race," and expressed "an unwillingness to even be near black people."
As TheWrap exclusively reported last month, Fox News fired longtime comptroller Slater last month after an internal investigation concluded she had engaged in a pattern of racist comments and behavior.
Also Read: Julie Roginsky Files Sexual Harassment Suit Against Fox News
Douglas also accuses Slater of mocking her for the "size of a breast that as removed as part of her cancer treatment." Douglas says Slater called her "boobs girl" or the "one-boomed girl."
Fox News issued the following statement to TheWrap: "We take complaints of this nature very seriously and took prompt and effective remedial action in terminating Judy Slater before Ms. Brown, Ms. Wright and Ms. Douglas sued in court and even before Ms. Wright and Ms. Douglas complained through their lawyer. There is no place for conduct like this at Fox News, which is why Ms. Slater was fired."
The amended version of the lawsuit also includes Fox News executive Dianne Brandi, who, according to the suit, once said that Slater would not be fired because she "knows too much" about improper behavior at the network.
Slater, who is white and worked at Fox for 19 years, was accused of asking one black employee if all three of her children were fathered by the same man, according to an individual familiar with the matter.
Also Read: Fox News Sued by 2 Black Women Charging 'Top-Down Racial Harassment'
In another incident, she responded to a good-night message from two black employees who stopped by her office by raising her hands in a "Hands up, don't shoot" gesture, a slogan associated with the Black Lives Matter movement, the source said.
Once employees came forward with complaints, Fox News Executive Vice President of Human Resources Kevin Lord led an investigation that resulted in Slater's termination.
The employees came forward with their complaints after the network started company-mandated sensitivity training in the wake of last summer's very public ouster of Fox News founder and CEO Roger Ailes over accusations of sexual harassment by anchor Gretchen Carlson and other female employees.
The 28 Most-Watched Cable News Shows of 2017 So Far, From Rachel Maddow to Tucker Carlson (Photos)
CNN, MSNBC and Fox News have all flourished in the Trump administration, but only one show can wear the crown as most-watched during the first quarter of 2017.
The key demographic that advertisers use to measure news programing is adults age 25-54, so that is how TheWrap ranks them to determine the most popular show.
Also Read: Fox News Has Best Quarter in Cable News History While CNN, MSNBC Show Growth
CNN/Fox News/ MSNBC
28. "CNN Newsroom"
CNN's midday news program averaged 249,000 viewers among the key news demo of adults age 25-54 during the first quarter of 2017.
27. "Anderson Cooper 360" Repeat
The late-night replay of "AC 360" averaged 252,000 viewers among the key demo.
Also Read: Sean Hannity Blasts CBS, Ted Koppel as 'Total Waste of My Time' (Video)
26. "Wolf"
The Wolf Blitzer newscast averaged 258,000 viewers between the ages 25-54 during the first three months of 2017.
25. "Hardball"
Chris Matthews' nightly show on MSNBC averaged 286,000 viewers among the key demo.
24. "The Lead with Jake Tapper"
Tapper has been praised for holding President Trump accountable, but doesn't crack the Top 20 cable news shows. He averaged 287,000 viewers among the key demo in the first quarter.
Also Read: Fox News Fires Comptroller Judy Slater, Accuses Her of Racist Comments and Behavior (Exclusive)
23. "Tucker Carlson Tonight" Repeat
The late-night repeat of "Tucker Carlson Tonight" averaged 319,000 viewers as the conservative commentator continues to succeed as Megyn Kelly's replacement on Fox News.
Also Read: 'The View' Co-Host Defends Sean Hannity: 'I Actually Do Think He's Good for America'
22. "Situation Room"
Wolf Blitzer's other daily CNN program averaged 320,000 viewers among the key demo, topping No. 26 "Wolf" despite the same host.
21. "11th Hour with Brian Williams"
Brian Williams' return to primetime has been a success, averaging 329,000 viewers among the key demo since the beginning of 2017.
Also Read: Here's Rachel Maddow's Plan to Beat Trump in 2020 (Exclusive)
20. "Shepard Smith Reporting"
Shepard Smith's program averaged 331,000 demo viewers and many Fox News fans have noticed that he has held President Trump accountable early in his administration.
19. "All In with Chris Hayes"
MSNBC's 8 p.m. ET host averaged 340,000 demo viewers to set up Rachel Maddow at 9 p.m.
Also Read: MSNBC's Rachel Maddow Played Us With Trump Taxes, But Will It Hurt Ratings?
18. "America's News HQ"
Fox News' weekend midday newscast averaged 343,000 viewers among the key demo.
Also Read: Judge Napolitano Stands By Wiretapping Claim in Return to Fox News (Video)
17. "Your World with Neil Cavuto"
The Fox Business Network star averages 348,000 viewers among the key demo when he heads to FBN's sister station, Fox News, for a daily program.
16. "Happening Now"
Jon Scott and Jenna Lee's weekday newscast on Fox News averaged 352,000 viewers among the key news demo.
15. "Erin Burnett Outfront"
CNN's "Outfront" averaged 367,000 demo viewers as Burnett continues to be among the most popular female anchors in cable news.
14. "Fox & Friends"
The show that is often praised by President Trump dominates cable news morning shows, averaging 376,000 demo viewers while MSNBC's "Morning Joe" averaged 182,000 and CNN's "New Day" picked up 205,000 in the same category.
Also Read: Twitter Goes Wild After President Trump Gets Caught Parroting Fox News
13. "CNN Tonight"
Don Lemon's nightly show is technically tied with No. 12 "Outnumbered," but we broke the tie by looking at total viewers, as opposed to the demo. Both programs averaged 381,000 demo viewers, but "Outnumbered" averaged 2 million total viewers compared to 1.1 million for "CNN Tonight."
Also Read: Happy New Year: Watch Don Lemon Get His Ear Pierced Live on TV, for Some Reason (Video)
12. "Outnumbered"
The Fox News midday show that features "One Lucky Guy" surrounded by four female co-hosts averaged 381,000 viewers to tie Don Lemon in the key demo.
11. "Last Word with Lawrence O'Donnell"
The former "West Wing" writer averages 392,000 demo viewers on his MSNBC show, almost cracking the Top 10.
Also Read: 'West Wing' Writer Lawrence O'Donnell Explains Why 2016 'Ruined' Political TV Drama
10. "America's Newsroom"
Bill Hemmer and Shannon Bream's newscast averaged 405,000 viewers among the key demo and recently made news when Judge Andrew Napolitano returned to Fox News Channel with an appearance on the show.
9. "The O'Reilly Factor" Repeat
The late-night replay of "The O'Reilly Factor" averaged 435,000 viewers among the key demo, proving how incredibly popular Bill O'Reilly remains.
Also Read: Bill O'Reilly Admits Criticism of Guest Falsely Billed as Swedish Official Is 'Valid'
8. "Anderson Cooper 360"
Cooper hosts CNN's most popular show, which averaged 470,000 viewers among the key demo.
Also Read: Anderson Cooper's Solution to Handling Trump: He Muted Him on Twitter
7. "The Five"
Fox News' "The Five" just moved into a fancy new studio and continues to be among the most-watched shows on cable, averaging 522,000 demo viewers.
6. "The Rachel Maddow Show"
MSNBC's most-watched show has been on fire and recently made headlines by releasing President Trump's 2005 tax return. Maddow averaged 527,000 viewers among the demo during the first quarter of 2017.
5. "Special Report"
Bret Baier's newscast on Fox News averaged 533,000 viewers among the demo, making it the most-watched straight news program in cable news.
Also Read: Fox News' Bret Baier Won't Tell You Who He's Voting For
4. "The First 100 Days"
Martha MacCallum's show might only be around for the first 100 days of Trump presidency, but it averaged 540,000 viewers to make her the most-watched woman on cable news.
Also Read: Martha MacCallum's 'The First 100 Days' Is a Hit – But What Happens on Day 101?
3. "Hannity"
Sean Hannity's Trump-friendly program has become must-see TV for fans of POTUS and the ratings are proof, as it averaged 668,000 viewers among the key demo.
2. "Tucker Carlson Tonight"
Megyn Kelly's replacement quickly became a fan favorite of Fox News supporters, averaging 689,000 viewers among the key demo.
Also Read: Short Hair, Don't Care: Why Tucker Carlson Is Fox News' Latest Star (Exclusive)
1."The O'Reilly Factor"
Bill O'Reilly continues to absolutely dominate the cable news landscape, averaging 728,000 demo viewers. When adding in the 8 p.m. ET airing with the late-night replay, "Factor" racks up more than 1 million viewers between the ages 25-54 every night.
Also Read: 2020 Presidential Candidates, Ranked by Vegas Odds (Photos)
CNN, MSNBC and Fox News thrive in Trump era

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Java API to convert TIFF or PNG or JPEG to SVG I have requirement  to convert TIFF or PNG or JPEG to SVG. Is it possible to convert to SVG. By converting the PNG/TIFF/JPEG to SVG, is there any chance to reduce the size?

A: Imagetracer is a free and open source (Public Domain) library and application which might be useful. Disclaimer: I made this.
Get ImageTracer.jar from
https://github.com/jankovicsandras/imagetracerjava
then you can use it from the CLI like this:
java -jar ImageTracer.jar input.png outfilename output.svg ltres 1 qtres 1 pathomit 16 colorsampling 1 numberofcolors 4 mincolorratio 0.02 colorquantcycles 3 scale 0.5 simplifytolerance 0 roundcoords 0 desc 0 blurradius 5 blurdelta 20
to get a posterized SVG from the input image, which is also scaled down to half the original size.
Raster to vector conversion does not generally reduce the file size by itself, especially with SVG output, because SVG is a verbose XML based text format without compression. But this is very dependent on the input image: a simple black and white logo with several megapixels resolution saved as BMP might be reduced from megabytes to kilobytes, but a colorful JPEG photo might go from tens of kilobytes to megabytes of SVG output.
You can test this also with Inkscape (or Potrace): the result SVG is usually bigger than the input image.

A: Take a look at Batik:
http://xmlgraphics.apache.org/batik/
It sounds like it should probably meet at least most of your requirements.  If there's some format Batik doesn't support, you can always get an additional library to read it into some "standard" bitmap format, then use Batik to convert it to SVG.

A: Do you want to wrap your raster images in an SVG container, or do you want to trace the rasters to vectors? If the latter then it's not a simple problem, but you can use a tracing library to do this. There are no guarantees that the file size will be reduced.

redpajama_set_name: RedPajamaStackExchange


text:
From: Innovation, Science and Economic Development Canada
Resources for Canadian businesses
Managing your business during COVID-19
Did you receive a letter from the Government of Canada? Please go to www.connecting-families.ca to sign up for the initiative.
Some notification letters were incorrectly dated February 2018 instead of February 2019. If you have received a letter dated February 2018, you are eligible to participate in this initiative.
The Connecting Families initiative is investing $13.2 million over five years, starting in 2017-2018 to help bridge the digital divide for Canadian families who may struggle to afford access to home Internet. With this funding, Computers for Success Canada has developed a secure online portal that will enable eligible Canadian families to access high speed Internet service packages for $10 per month from participating Internet Service Providers.
The Connecting Families initiative will help connect up hundreds of thousands of Canadians to the Internet and will distribute up to 50,000 computers to eligible households. These efforts will ensure that more Canadian families and youth have access to the valuable resources available on the Internet and give them access to the tools they need to be successful and thrive.
In Budget 2017, the Government of Canada introduced its Innovation and Skills Plan, which focuses on people and addresses the changing nature of the economy. In order to keep pace with these changes, it is important to equip Canadians with the tools, skills, access and experience they need to fully participate in the digital economy.
The Connecting Families initiative is a part of a suite of digital skills, accessibility, and Internet affordability programs that aim to bridge Canada's digital divide. This initiative will enable and encourage the participation of more Canadians in the digital world, including those not fully comfortable with digital technology, so they too can benefit from innovation and the opportunities it brings.
The importance of access
Daily aspects of Canadians' lives are increasingly touched by digital technology, and access to high-speed Internet has become an essential service and a key driver for improving our economic and social well-being.
More than ever, Canadians need access to fast, reliable and affordable Internet to learn, work, socialize and access Government of Canada services. Many schools require that students access Internet resources for homework projects, post-secondary institutions often provide learning and course materials online, families and friends are increasingly socializing through web-based communication tools, and the Internet is being used to search for jobs, access government services, book important appointments and find new information.
Despite the importance of high-speed Internet, almost half of Canadian households with an annual income of $30,000 or less do not have access. The Connecting Families initiative brings together public, private and non-profit sectors to ensure that more Canadian families have access to the Internet.
Do you have questions? Contact us.
Frequently asked questions: Connecting Families
Participating Internet Service Providers
Page: E111-H00000