text:
\section{INTRODUCTION}
Reliable and long-range obstacle detection is essential to enable low-altitude flights or landing maneuvers for fixed-wing unmanned aerial vehicles (UAVs).
While full size drones often employ precise Lidar or Radar systems, small-scale UAVs usually cannot afford to carry this type of heavy payload with high power consumption, but need to rely on cheaper and more light-weight optical camera systems.
To recover the absolute scale information from cameras, either a classical stereo setup with a fixed and calibrated baseline transformation is used, or virtual stereo pairs are computed from a monocular camera setup.
\subsection{Monocular Camera Setup: Flying into the Epipole}
Monocular visual-inertial sensor setups have proven to be well suited for applications such as simultaneous localization and mapping (SLAM) \cite{Klein2007} or planar dense reconstruction \cite{Fusiello2000}.
The algorithms perform best when using a down-looking camera where the epipoles are outside of the camera field of view and fronto-parallel motion with respect to the ground is performed.
However, within the application of obstacle detection and avoidance, the region of interest is in front of the aircraft and a forward facing or oblique camera needs to be employed.
In this setup, the optical axis is closely aligned with the aircraft's direction of flight, resulting in a blind spot just in the region of interest.
While this problem can be partially overcome by using, for instance, polar rectification \cite{Pollefeys1999}, the procedure is inherently error prone since the pixels close to the image center have only small changes in image coordinates in the subsequent frame.
\begin{figure}[t]
\includegraphics[width=\linewidth, trim=44 8 0 20,clip=true]{tikz/teaser.pdf}
\caption[Caption for LOF]{Envisioned use-case for the proposed non-rigid wide-baseline stereo vision algorithm for fixed-wing UAVs.\footnotemark \ Bottom left: One of the two light-weight, rigid camera-IMU rigs. Bottom right: Characteristic frame from synthetic dataset comparing depth map obtained from ground-truth poses, estimated poses, and poses obtained from a fixed baseline assumption.}
\label{fig:teaser}
\end{figure}
\footnotetext{A video illustrating \emph{AtlantikSolar}'s flapping wing behavior is accessible under \url{https://youtu.be/8m76Mx9m2nM}.}

\subsection{Fixed-Baseline Stereo}
Due to the shortcomings of the monocular sensor setup,  rotary-wing UAVs, such as quadro- or hexacopters, are usually equipped with a fixed baseline stereo pair to perform onboard and real-time obstacle avoidance while providing a high level of reliability.
However, the baseline of the stereo rigs, and hence the range where low depth uncertainties are obtained, are optimized for indoor scenarios with maximum speeds of few meters per second.
The small baselines, ranging from $\unit[9]{mm}$ up to $\unit[0.5]{m}$ \cite{Warren2016}, leave not enough time for short-term path planning, control, and actuators to react in highly dynamic fixed-wing UAV scenarios.
Upscaling the baseline to more suited depth ranges while still ensuring rigidity of the stereo rig, on the other hand, would impair aerodynamics, the required flexibility of the wing, and inevitably lead to a higher payload mass.
\subsection{Aim of this Work: Non-Rigid Wide-Baseline Stereo}
Consequently, for fixed-wing aircrafts it seems appealing to take advantage of the wing span and to mount the cameras on the outer wing region.
However, especially solar-powered aircrafts \cite{Oettershagen2017} which are optimized for wing area and updraft, show flapping wing behavior leading to translational and rotational deformations  from the nominal baseline in the range of several degrees and centimeters.
These disturbances affect the depth estimation in two ways:
Firstly, since the assumption of an accurately calibrated fixed relative transformation between the stereo pair is violated, the depth estimates become heavily distorted.
For example, disturbances in yaw and pitch angle estimates lead to errors in the depth estimate which are quadratic to the depth \cite{Dang2009}.
Secondly, stereo vision algorithms usually rely on matching correspondences along epipolar lines \cite{Hartley2004}.
Therefore, even small errors in the estimated rotation can already be fatal since the correspondences are searched along the wrong line.
This leads to an almost empty depth map since no correspondences can be found.
It could be overcome by a full 2D search over the whole image which, however, is not computationally feasible in real-time.

The aim of this work is therefore to estimate the time-varying relative pose between the two camera-IMU rigs with low delay and close to the image capture frequency as illustrated in Fig. \ref{fig:teaser}.
We achieve this by fusing the high-frequency, low-variance inertial sensors with the low-frequency, bias-free vision-based relative pose estimation in combination with the probabilistic wing model in an efficient Extended Kalman Filter (EKF) formulation \cite{Achtelik2011,Achtelik2014}.

In summary, we present a light-weight formulation for wide-baseline non-rigid stereo that builds up on \cite{Achtelik2011}.
Compared to \cite{Achtelik2011}, we see the following contributions:
Most importantly, we propose a wing model in form of a relative pose prior.
While \cite{Achtelik2011} makes no assumption about the relative pose of the IMUs, we take advantage of our wing deflection model to better constrain the EKF and to identify and reject visual outliers.
While \cite{Achtelik2011} treated the vision module as a black box, we extract and match features and incorporate a relative perspective n-point (PNP) solver as vision-based relative motion estimation module.
The pipeline is validated extensively based on synthetic datasets, in particular, with respect to the quality of a) the estimated baseline transformation, and b) the resulting depth map.
\section{RELATED WORK}
The \textit{auto-calibration} problem is the retrieval of the rigid, non-changing stereo baseline transformation in known \cite{furgale2013unified} or unknown \cite{warren2013online,hansen2012online} environments.
In the latter, images are taken over time and keypoints are extracted to perform stereo bundle adjustment.
This can be extended towards \textit{re-calibration} of stereo rigs during operation, also known as \textit{online-calibration}.
By having a good initial estimate, e.g.\@\xspace obtained from auto-calibration, the baseline can be re-estimated assuming it only changed slightly.
The techniques are then essentially the same as for auto-calibration in unknown environments.
Perturbations from the nominal stereo baseline can be detected by an increase in reprojection errors, hence triggering a re-calibration process.
Extending further on this idea, Warren et al. \cite{warren2013online} continuously estimate slow changes (e.g.\@\xspace thermally induced) in the baseline.
The key difference of the \emph{continuous self-calibration} methods are that they do not assume that the baseline is rigid over the whole duration of the calibration process.
However, in contrast to our setting, the baseline changes in \cite{warren2013online} are still assumed to be only small and slow.
Work on continuous calibration in presence of high-frequent noise exists.
In \cite{warren2016long}, a down-looking stereo pair with a wide-baseline (\SI{0.7}{m}) is employed and the relative transform between the cameras, together with the poses of the stereo rig itself is estimated offline in a bundle adjustment problem.
Similarly to our work, Warren et al. incorporate prior knowledge of the deformation in the stereo rig.
This is achieved in form of a cost-based bound to tightly constrain the estimated transform.
However, in \cite{warren2016long}, the anticipated changes in baseline are induced by vibration and hence relatively small (a few millimeters in translation and only tenth of a degree in rotation) making the prior a good assumption.
In contrast, we are dealing with an increase of two order of magnitudes (decimeters and more than ten degrees).\newline
We are aware of two different approaches to real-time baseline estimation in which the baseline deviations closer resemble our scenario:
The \emph{first approach}, described in \cite{lanier2011modal, lanier2010, Short2009a}, is based on a modal-analysis of a wing which enables to estimate vibrational disturbances by only measuring with accelerometers (and no gyroscopes).
In \cite{lanier2011modal}, this method is introduced and show-cased on an Euler-Bernoulli beam model employing two accelerometers.
This is extended in \cite{lanier2010} to wing-mounted stereo rigs and experimental results are obtained during periodic and random excitation of a real wing model.
A total of 24 accelerometers are mounted over the wing to perform modal analysis.
The camera mount-points are on the wing-tips resulting in a theoretical baseline of \SI{240}{cm}.
However, only an analysis of the wing and the corresponding baseline estimation errors was performed.
The results suggest significant improvements over a fixed-baseline assumption.
Note that these results are obtained in experiments performed in small scale and then extrapolated.
In contrast, we mount the IMUs (including a gyroscope) spatially close and rigidly to the cameras and do not rely on a physically motivated wing-model.
Furthermore, their approach suggests taking images when the motion is minimal to minimize effects of motion blur.
Instead we rely on global shutter cameras with short and synchronized shutter times.

The \emph{second approach} is described in \cite{Achtelik2011}:
It employs an EKF to estimate the relative position between two micro air vehicles (MAV), each equipped with an IMU and a down-looking camera, assuming an overlapping field of view.
The authors leave the visual part as a black box and suggest the use of a visual SLAM framework with scale propagation such as PTAM \cite{Klein2007}.
In contrast to our work, since MAVs can move independently of each other, no prior knowledge on the baseline transform is assumed.
Our work is based to a large extent on the EKF described in \cite{Achtelik2011}, but extends the measurement model by fusing our prior knowledge of the baseline. Furthermore, the vision module is implemented in form of a relative PNP solver.
Since the latter is based solely on sparse correspondence matching of the current stereo pair, no scale propagation can be performed which simplifies the EKF.
Furthermore, as the relative baseline transformation may change quickly during flight, no feature tracking or descriptor matching over time is performed.
\section{METHODOLOGY}
An overview of our proposed framework is illustrated in Fig. \ref{fig:overview}.
The incoming image stream is used both, for visual estimation of the relative pose using a relative PNP solver, and for generating depth maps.
The visual estimates are fused with the baseline prior and fed into an EKF together with high-frequent ($\geq 100$ Hz) IMU measurements.
The obtained filtered estimate of the  baseline transformation $\hat \vT$ is then used to rectify the images so that dense matching can be done reliably and fast via correspondence search along epipolar lines \cite{hartley2003multiple}.
The resulting depth maps can be fed into external applications, e.g.\@\xspace a flight controller to avoid obstacles, or to reconstruct the environment.
We adopt the EKF as described in \cite{Achtelik2011}.
There is one fundamental modification:
As our visual pose estimator is based solely on the most recent two frames and does not have scale propagation we do not include $\lambda$ in our state vector.
Correspondingly, our measurement model for obtaining relative pose estimates is adopted.
For completeness and to clarify the modifications with respect to \cite{Achtelik2011}, we summarize all elements of the Kalman filter that are required for understanding in the following.
\begin{figure}[bth]
\includegraphics[width=1\linewidth]{arxiv_images/overview}
\caption{Proposed framework to estimate the time-varying relative transformation between two cameras. The efficient EKF formulation fuses a wing deformation prior with the information from two IMUs and two cameras.}
\label{fig:overview}
\end{figure}
\subsection{The State Vector}
The state consists of the relative angular rates $\vom_1^1$, $\vom_2^2$, the linear accelerations expressed in the world frame $_w \va_1^1$, $_w \va_2^2$, the relative orientation expressed as a quaternion $\bar q_1^2$ and the (metric) relative position $\vp_1^2$ and finally the relative velocity $_w \vv_1^2$.
Stacked together these values form the 22-element state vector
\begin{equation}
 \vx = \left[ \mbox{$\bar q_1^2$}^\top \; {\vom_1^1}^\top \; {\vom_2^2}^\top \; {\vp_1^2}^\top \; {_w \vv_1^2}^\top \; {_w \va_1^1}^\top \; {_w \va_2^2}^\top \right]^\top.
\end{equation}
To simplify notation, super- and subscripts are dropped in the following and the states are simply denoted by
\begin{equation}
 \vx = \left[ {\bar q}^\top \; \vom_1^\top \; \vom_2^\top \; \vp^\top \; \vv^\top \; \va_1^\top \; \va_2^\top \right]^\top.
\end{equation}
\subsection{State Equations}
Angular velocities $\vom_1, \vom_2$ and linear accelerations $\va_1$, $\va_2$ are modeled as independent random zero-mean Gaussian walks with (diagonal) covariances $\vsi_{\vom_1}, \vsi_{\vom_2}, \vsi_{\va_1}$ and $\vsi_{\va_2}$.
The corresponding random vectors are denoted by $\vn_{\vom_1}$, $\vn_{\vom_2}$, $\vn_{\va_1}$ respectively $\vn_{\va_2}$.
For quaternion multiplication the angular velocities are represented as quaternions $\bar \omega_i = [0 \; \vom_i^\top]^\top$ (for $i = 1,2$), $\vC$ denotes the rotation matrix corresponding to the quaternion $\bar q$ and $\lfloor \vom_1 \times \rfloor$ denotes the skew-symmetric matrix.
The final state equations are as follows:
\begin{align}
\dot{\bar q} & = 0.5 \cdot \left({\bar q} \otimes \bar \omega_2 - \bar \omega_1 \otimes \bar q \right) \label{update:q} \\
\dot\vom_1 & = \vn_{\vom_1} \\
\dot\vom_2 & = \vn_{\vom_2} \\
\dot\vp & = \vv - \lfloor\vom_1 \times \rfloor \cdot \vp \label{update:p} \\
\dot\vv & = \vC \cdot \va_2 - \va_1 - \lfloor \vom_1 \times \rfloor \cdot \vv \label{update:v} \\
\dot\va_1 & = \vn_{\va_1} \\
\dot\va_2 & = \vn_{\va_2}
\end{align}
Detailed derivations can be found in \cite{Achtelik2011}.
\subsection{Error State Representation}
\label{ekf:errstate}
From here on, an estimated state is denoted by $\hat .$ and a preceding $\Delta$ denotes the error state for an additive error.
For the quaternion $\bar q$ a multiplicative error model is used.
The error is denoted by $\delta\vth$.
The error state vector is then
\begin{equation}
\tilde \vx = \left[ \delta\vth^\top \; \Delta\vom_1^\top \; \Delta\vom_2^\top \; \Delta\vp^\top \; \Delta\vv^\top \; \Delta\va_1^\top \; \Delta\va_2^\top \right]^\top.
\end{equation}
Again, we refer to \cite{Achtelik2011} for derivations of the error state update equations.
Noting that $\hat \vC$ is the corresponding rotation matrix of the quaternion $\hat \vq$, the final error state equations are
\begin{align}
\dot{\delta\vth} = &\ -\lfloor \hat\vom_2 \times \rfloor \cdot \delta\vth - {\hat\vC}^\top \cdot \Delta\vom_1 + \Delta\vom_2\\
\dot{\Delta\vom_1} = &\ \vn_{\vom_1} \\
\dot{\Delta\vom_2} = &\ \vn_{\vom_2} \\
\dot{\Delta\vp} = &\ \lfloor \hat\vp \times \rfloor \cdot \Delta\vom_1 - \lfloor\hat\vom_1\times\rfloor\cdot\Delta\vp + \Delta\vv \\
\dot{\Delta\vv} = &\ -\hat\vC\cdot\lfloor\hat\va_2\times\rfloor\cdot\delta\vth + \lfloor\hat\vv\times\rfloor\cdot\Delta\vom_1 \nonumber \\
   &\ - \lfloor\hat\vom_1\times\rfloor\cdot\Delta\vv - \Delta\va_1 + \hat\vC\cdot\Delta\va_2 \\
\dot{\Delta\va_1} = &\ \vn_{\va_1} \\
\dot{\Delta\va_2} = &\ \vn_{\va_2}
\end{align}
\subsection{State Covariance Prediction}
These equations enable the computation of the continuous system matrix $\vF_c = \frac{\partial\dot{\tilde\vx}}{\partial\tilde\vx}$ and noise matrix $\vG_c = \frac{\partial\dot{\tilde\vx}}{\partial\vn}$ with $\vn = \left[ \vn_{\vom_1}^\top \; \vn_{\vom_2}^\top \; \vn_{\va_1}^\top \; \vn_{\va_2}^\top \; \right]^\top$.
The computed Jacobi matrices are given in \cite[(4.21)]{Achtelik2014}.
Assuming $\vF_c$ to be constant over the integration period yields $\vF_d = \exp(\Delta t \cdot \vF_c)$ for a given time-step $\Delta t$.
This is approximated to the zeroth order term by expanding the exponential series, resulting in $\vF_d \approx \vI + \vF_c\Delta t$.
With the continuous time noise covariance matrix $\vQ_c = \diag\left(\left[\vsi^2_{\vom_1} \; \vsi^2_{\vom_2} \; \vsi^2_{\va_1} \; \vsi^2_{\va_2}\right]\right)$, the discrete time noise covariance matrix $\vQ_d$ is computed according to \cite{Maybeck1979}.
Assuming $\vF_d$ to stay constant during the integration period yields
\begin{align}
\vQ_d & = \int_0^{\Delta t}\vF_d(\tau) \cdot \vG_c \cdot \vQ_c \cdot \vG_c^\top \cdot \vF_d(\tau)^\top \cdot d\tau \\
      & = \Delta t \cdot \vF_d \cdot \vG_c \cdot \vQ_c \cdot \vG_c^\top \cdot \vF_d^\top.
\end{align}
The updated state covariance matrix for the EKF is then computed as
\begin{align}
\vP_{k+1 | k} = \vF_d \cdot \vP_{k|k}\cdot\vF_d^\top + \vQ_d.
\end{align}
\subsection{State Prediction}
The states are predicted by zeroth order integration according to (\ref{update:q}), (\ref{update:p}), respectively (\ref{update:v}) given above:
\begin{align}
\hat{\bar q}_{k+1} & = \hat{\bar q}_{k} + 0.5 \cdot \Delta t \cdot (\hat{\bar q}_k \otimes \hat{\bar\omega}_{2,k} - \hat{\bar\omega}_{1,k} \otimes \hat{\bar q}_k) \\
\hat\vp_{k+1} & = \hat\vp_k + \left(\hat\vv_k - \lfloor\hat\vom_{1,k}\times\rfloor\cdot\hat\vp_k\right)\cdot\Delta t\\
\hat\vv_{k+1} & = \hat\vv_k + \left(\hat\vC\cdot\hat\va_{2,k} - \hat\va_{1,k} - \lfloor\hat\vom_{1,k}\times\rfloor\cdot\hat\vv_k\right)\cdot\Delta t
\end{align}
\subsection{Vision-Based Relative Pose Measurements}
The vision-based relative pose estimates are obtained in three steps.
Classical feature descriptor, matcher, and relative PNP types are selected for a proof of concept:
Firstly, SURF keypoints \cite{Bay2006} are detected and their descriptors are extracted in both images.
Secondly, feature correspondences are established using a FLANN-based matcher \cite{Muja2009}.
Finally, the bearing vectors are computed for both frames and every matched feature.
Based on the two sets of bearing vectors, a relative pose up to a scale is estimated by means of finding the fundamental matrix $\vF$ \cite{Hartley2004}.
The estimated fundamental matrix $\vF$ is then converted to a unit vector $\hat\vp_v$ representing the direction of translation and the rotation quaternion $\hat{\bar{q}}_v$.
To solve the PNP problem, we employ the 5-point Nister algorithm \cite{Nister2004}.
Since PNP solvers are sensitive to wrong associations, Random Sample Consensus (RANSAC) \cite{Fischler1981} is employed \cite{Kneip2014}.
To fuse the visual estimates, the output of the algorithm is interpreted probabilistically:
The measurements are modeled as
\begin{align}
\hat \vp_v &= \frac{\vp + \Delta\vp_v}{\left\| \vp + \Delta\vp_v \right\|_2}  \\
\hat{\bar q}_v &= \bar q \otimes \delta\bar q_v
\end{align}
where $\hat\vp_v$ and $\hat{\bar q}_v$ are the direction of translation respectively orientation estimated by the vision-based relative pose estimation module (i.e. in our case the PNP solver). The corresponding measurement error is denoted by $\Delta\vp_v$, $\delta\bar q_v$.
 %
Approximating the error quaternion $\delta\bar q_v$ by small angles $\delta\vth_v$ allows expressing the error transformation as a 6-dimensional vector $\delta\vT_v = \left[ \delta\vth_v^\top \; \Delta\vp_v^\top \right]^\top$.
This error is then approximated by a zero-mean Gaussian with covariance $\vSi_v$.
\subsection{Establishing a Wing Model}\label{baseline-prior}
In this section, the notion of baseline calibration is extended beyond rigidity.
Instead of finding the fixed relative transform between the two cameras of a stereo rig as the result of an (usually over-constrained) optimization problem flexibility is embraced.
The wing model is captured by a nominal baseline transformation $\vT_\mu$ and a probabilistic error model captured in a random vector $\delta\vT$.
For notational clearance the baseline transform $\vT_\Ione^\Itwo$ is simply denoted by $\vT$ or in its parts $\bar q$ and $\vp$.
The mean transform $\vT_\mu$ is expressed in parts by the mean quaternion $\bar q_\mu$ and the mean relative position $\vp_\mu$.
This allows to express the disturbance in rotation via a multiplicative error model $\bar q = \bar q_\mu \otimes \delta\bar q$ and in position as a simple additive error $\vp = \vp_\mu + \Delta\vp$.
The (small) quaternion $\delta \bar q$ is then approximated via the small angle approximation $\delta\vth$.
With that, $\delta \vT = \left[ \delta\vth^\top \; \delta\vp^\top \right]^\top$ is defined.
Note that the random vector $\delta \vT$ has zero mean.
In the spirit of the EKF, the error of the \textit{calibration} is modeled as a Gaussian: $\delta \vT \sim \mathcal{N}(\bm 0, \vSi_c)$.
The six dimensions of $\delta \vT$ are correlated (i.e. when there is a roll-disturbance, there is also a disturbance in the z direction), however not linearly.
Therefore, the distribution is approximated as independent Gaussians and thus the covariance matrix is diagonal: $\vSi_c = \diag(\vsi_{\delta\vT}^2)$.
\subsection{Fusing the Baseline Prior with Visual Estimates}
\label{combpos}
The visual measurements of the baseline $\hat{\bar q}_v$ and $\hat \vp_v$ are then combined with the baseline prior to obtain the maximum a posteriori estimates $\hat{\bar q}_f$ and $\hat \vp_f$.
Since both, the baseline prior and the measurement error, are modeled as zero-mean Gaussians, the a posteriori distribution is again a Gaussian.

The visual estimates are expressed as the deviations $\delta\hat{\bar q}_v$ from the calibrated mean such that $\hat {\bar q}_v = \bar q_\mu \otimes \delta\hat{\bar q}_v$ and $\Delta\hat\vp_v = \hat \vp_v - \vp_\mu$.
The error quaternion $\delta\hat{\bar q}_v$ is approximated by small angles $\delta\hat\vth_v$.

The new estimates of the baseline deviation $\delta\hat{\bar q}_f$ and $\Delta\hat\vp_f$ are computed such that the estimated baseline is $\hat{\bar q}_f = \bar q_\mu \otimes \delta\hat{\bar q}_f$ and $\hat\vp_f = \vp_\mu + \Delta\hat\vp_f$.
Again, $\delta\hat{\bar q}_f$ is approximated by small angles $\delta\hat\vth_f$.
By interpreting the fusion of the measurement with the prior as a KF update step, the new estimates and the a posteriori covariance matrix is given by:
\begin{align}
\begin{bmatrix}\delta\hat\vth_f \\ \Delta\hat\vp_f\end{bmatrix} & = \vSi_c(\vSi_c + \vSi_v)^{-1}\begin{bmatrix}\delta\hat\vth_v \\ \Delta\hat\vp_v \end{bmatrix} \\
\vSi_f & = \vSi_c - \vSi_c(\vSi_c + \vSi_v)^{-1}\vSi_c
\end{align}
\subsection{Visual Outlier Rejection}
The baseline prior is used to filter outlier estimates from the relative PNP pipeline.
Depending on the scene, the visual pose estimation may give severe outliers.
This happens, for instance, if the cameras see mostly uniformly colored sky or landscape without salient keypoints for detection and matching.
In these cases, the visual pose estimate is ignored altogether and, instead, an artificial measurement update corresponding to the baseline prior is fed into the EKF.
Filtering of outliers is achieved by limiting the maximum distance of the visual estimate $\hat\vT_v$ to the mean baseline $\vT_\mu$ in any dimension by a factor $k$ (in our case 2) of the standard deviation.
That is, we set $\hat{\bar q}_f = \hat{\bar q}_\mu$ and $\hat \vp_f = \hat \vp_\mu$ if $\left\|\vSi_v^{-1}\left[\delta\hat{\vth}_v^2 \; \Delta\hat{\vp}_v^2\right]^\top\right\|_\infty > k^2$.
\section{Simulation Environment}
\emph{Gazebo}-based \textit{RotorS} \cite{furrer2016rotors} is used to simulate the forces acting on the flexible wings, and to collect precise ground truth poses and IMU measurements.
The camera poses and a publicly available mesh are imported into \textit{Blender} and the photo-realistic images are rendered.
\begin{figure}[htb]
	\includegraphics[width=\textwidth]{arxiv_images/sim_blender_opt.pdf}
\centering
\caption{Synthetic dataset: Overview mesh and aircraft trajectory visualized in \emph{Blender}. The image in the bottom right corner shows the aircraft with two flexible wings modeled in \emph{Rotors}. The mesh was downloaded from \url{https://skfb.ly/Sq7J}.}
\label{fig:world}
\end{figure}
\paragraph{Aircraft with flexible wings} The aircraft is modeled by two wings, each connected to the airframe by a joint.
The joints permit only roll motion, i.e.\@\xspace rotational movement around the aircraft's body's $\vx$-axis.
Furthermore, as shown in Fig. \ref{fig:wing}, each wing in itself is modeled by two rigid bodies connected by another joint permitting only movement around the wing's center line, i.e.\@\xspace pitching movements.
Both joint angles are controlled by a PD-controller which emulates a spring-damper system.
The total mass of the aircraft is \SI{2.8}{kg}, each wing has a mass of \SI{0.4}{kg}.
\begin{figure}[htb]
\centering
\includegraphics[width=\linewidth]{arxiv_images/wing}
\caption{View from back and right on the flexible
wing modeled by two rigid bodies.
There are two degrees of freedom: the roll angle $\alpha$ and the pitch angle $\beta$.}
\label{fig:wing}
\end{figure}
A disturbance force $\vF_\text{dist} = \vF_p + \vF_r$ is applied on the wing tips.
The disturbance force is composed of a periodic force $\vF_p = a_p \sin\left(f_p t\right)$ with magnitude $a_p$ and frequency $f_p$ and a random force $\vF_r$ of random magnitude $a_r \sim\mathcal{N}(a_r, \sigma_{a_r})$ applied at fixed frequency $f_r$ for a duration of $t_r$.
The random forces simulate the effect of wind gusts and excite the wing into brief non-uniform oscillations.
The sinusoidal force is applied in same phase with a magnitude of $\SI{0.25}{N}$ and frequency of $f_p = \SI{1.5}{Hz}$ on both wing tips.
The random magnitude is sampled from $\mathcal{N}(1.0, 0.1)$ every $\SI{8}{s}$ (i.e. $f_r = \SI{0.125}{Hz}$), however, independently for each side and applied over a duration of $\SI{0.4}{s}$.
The aircraft follows the predefined trajectory as depicted in Fig. \ref{fig:world}.
A 6-DoF PID-controller is used which applies forces along and torques around all three body axes based on the desired velocity and position.
\paragraph{Visual-Inertial Camera Rig}
\begin{figure}[tb]
\raggedleft
\setlength\figureheight{6em}
\setlength\figurewidth{\linewidth - \widthof{$-10$} - 20pt}
\input{tikz/stereo-rig.tex}
\caption{The FOV in the $x$-$z$-plane of the left (red) and right (green) camera with a baseline of \SI{3}{m}.
In the overlapping FOV the area corresponding to each disparity value (12 to 140) is colorized uniformly.}
\label{fig:stereo-rig}
\end{figure}
The geometry of the visual-inertial stereo rig with a nominal baseline of \SI{3}{m} is depicted in Fig. \ref{fig:stereo-rig}.
The cameras\footnote{Perfect pinhole cameras without distortion and a resolution of $720 \times 480$ pixels ($\unit[0.36]{MP}$).} are rotated towards each other by \SI{8}{\degree} to increase the overlapping field of view (FOV).
The IMU coordinate frames coincide with the camera frames, their axes are kept parallel.
The IMU \textit{ADIS 16448} \cite{adisIMU} is simulated with white noise variances of \SI{1.225e-7}{rad/s} for angular velocities and \SI{1.6e-5}{m^2/s} for linear accelerations.
\paragraph{Initialization of EKF}
The baseline prior is obtained by empirically estimating the mean and variances of the baseline transform as described in Section \ref{baseline-prior}.
To simulate an imperfect baseline-prior calibration the variances are increased by $\unit[10]{\%}$.
The mean and standard errors of the real relative transform are shown in Table \ref{table:gt}. The mean rotation is parametrized as roll-pitch-yaw angles $\vmu_{\bm{\theta}}$.
\begin{table}[htb]
\footnotesize
\begin{tabular}{|l|l|l|l|l|}\hline
\rowcolor{black!10} &  $\vx$ & $\vy$ & $\vz$ & unit \\ \hline
$\vmu_{\bm{\theta}}$          & -0.51   & -1.8e-6 & -3.3e-4 & deg \\ \hline
$\vsi_{\delta\vth}$ &   1.9    &  7.1e-3 &  1.3e-2 & deg \\ \hline
$\vmu_{\bm{p}}$           &-0.0087 & -3000.0 &  13.4  & mm\\ \hline
$\vsi_{\Delta\vp}$  &   0.27   &  3.0    &  50.5  & mm\\ \hline
\end{tabular}
\caption{True nominal baseline transform used in the synthetic dataset.}
\label{table:gt}
\end{table}
\section{Simulation Results}
\subsection{Evaluation of Estimated Camera Poses}
In this section, we compare the camera pose estimates obtained from a) using the fixed calibration only, b) using joint prior-visual estimates only (with five-point Nister algorithm), c) using the prior calibration and IMU measurements only and finally e) using the full pipeline (prior, vision, and IMU).
Note that the EKF fused with IMU measurements only (without baseline prior) diverges quickly and is not shown in the following.
Fig. \ref{fig:box} visualizes the individual normalized root mean squared errors (RMSE) with respect to the ground truth poses.
There are two main observations: Firstly, incorporating the IMU decreases the errors significantly.
Secondly, since the RMSE of the relative PNP solution is relatively high, fusing them into the EKF reduces the overall RMSE only slightly.
\begin{figure}[htb]
\raggedleft
\setlength\figureheight{1.6cm}
\setlength\figurewidth{\linewidth - \widthof{visual + Prior} }
\input{plots/compare.tex}
\caption{Normalized RMSE of a fixed baseline assumption, fused estimate of visual pose estimation with the baseline prior, using IMU and baseline prior, and the full pipeline (visual, IMU, and baseline prior).
The dashed line shows the mean.}
\label{fig:box}
\end{figure}
\begin{table}[htb]
\footnotesize
\centering
\begin{tabular}{|l|l|l|l|l|l|l|} \hline
\rowcolor{black!10}  &$\delta\tilde\vth_x$ & $\delta\tilde\vth_y$ & $\delta\tilde\vth_z$
& $\Delta\tilde\vp_x$ & $\Delta\tilde\vp_y$ & $\Delta\tilde\vp_z$  \\ \hline
Fixed    & 1.96  & 0.0071 & 0.0102 & 0.269 & 3.06   & 51.2 \\ \hline
IMU + Prior & 0.083 & 0.0070 & 0.0095 & 0.375 & 2.83   & 14.7 \\ \hline
\end{tabular}
\caption{Rotational RMSE in $\unit[]{deg}$ and translational RMSE in $\unit[]{mm}$.}
\label{table:est}
\end{table}
\normalsize
As can be seen in Table \ref{table:est}, the rotations around the $\vy$- (pitch) and $\vz$-axis (yaw) as well as the motions along $\vx$- and $\vy$-axis have similarly small deviations as our error estimate and hence do not significantly affect correspondence matching or depth estimation.
For the rotation around $\vx$-axis we see an improvement of over one order of magnitude (from around \SI{2}{\degree} to \SI{0.08}{\degree}) and in movement along $\vz$-axis a three-fold improvement.
Rotational disturbances affect the matching process much stronger than translational disturbances, especially since we deal with objects far away.
Given our large baseline and long distances, the translational deviations on $\vy$- and $\vz$-axis do not have a significant impact.
For instance, the disturbance of around \SI{15}{mm} on $\vz$-axis, which corresponds to the optical $\vy$ axis, does not result in any pixel disturbance for points that are \SI{13}{m} or further away.
\begin{figure}[tb]
\raggedleft
\setlength\figureheight{2cm}
\setlength\figurewidth{\linewidth - 2.8em}
\input{plots/timeseries.tex}
\caption{RMS errors plotted over the first \SI{40}{s} in orientation and position of estimated deviations based on the fixed baseline assumption or using the proposed pipeline.
The random disturbance force results in high peaks every \SI{8}{s} while the periodic disturbance results in high-frequent errors.
The horizontal black line represents the standard error of the disturbance.}
\label{fig:est-time}
\end{figure}
\begin{figure}[htb]
\raggedleft
\setlength\figurewidth{\linewidth - 2.8em}
\setlength\figureheight{2.4cm}
\input{plots/imu-noise.tex}
\caption{Normalized RMSE versus noise-level expressed as multiples of the noise variances of the \textit{ADIS 16448} IMU. Note the logarithmic y-axis.}
\label{fig:noise}
\end{figure}
Comparing to the fixed baseline, which suffers both from the periodic, regular disturbances, and the low-frequent random disturbances, our proposed approach performs significantly better as further shown in Fig. \ref{fig:est-time}.
The first peak at \SI{4}{s} in positional error is due to the initialization of the EKF.
Relative velocity, angular velocities, and linear accelerations are all initialized to zero.
Position and orientation are initialized using the baseline prior.
However since our trajectory starts mid-air, our initialization values for the EKF are incorrect, especially for the linear accelerations.
This leads to relatively bad early estimates in position of up to about \SI{5}{s}, until the acceleration values and thus velocity estimates converge.
This behavior is less pronounced in orientation since the angular velocities are directly measured.
The small peak at around \SI{17}{s} can be explained by an exceptionally large random disturbance at this time step.
Since our IMUs operate at only \SI{100}{Hz} not all acceleration spikes can be captured correctly.
This leads to wrong acceleration values similar to the wrong initialization case.

In a next step the sensitivity of our approach with respect to IMU noise is analyzed.
Fig. \ref{fig:noise} plots the obtained normalized RMS errors for the proposed pipeline.
We express the noise levels as multiples of the noise of the \textit{ADIS 16448} IMU.
Clearly, increasing noise affects the estimates, but for up to about an eight-fold increase in noise the estimates still stay relatively close.
From this experiment, we conclude that even in real-world scenarios our approach is expected to yield good results.
\subsection{Evaluation of Depth-Maps}
Finally, we wish to support our claim that the resulting relative pose estimates are of such a high quality that accurate depth maps can be obtained.
For comparison, three depth maps are generated at each timestep: one based on the ground-truth transform $\vT$, one based on our estimate $\hat \vT$, and one based on a fixed baseline transformation, denoted by $\vT_\mu$.
The disparities of dense correspondences in the rectified stereo images are computed using Stereo Block Matching (BM) \cite{bradski2000opencv}.
Given the disparity $d$ of a pixel in the rectified images, the depth of a pixel is given by $z = f\cdot b / d$ where $f$ is the focal length and $b$ the baseline of the rectified stereo pair.
The obtained depth map image from transform $\vT$ is denoted by $\vD(\vT)$.
First, the performance of our approach is qualitatively demonstrated based on three characteristic frames in Fig. \ref{fig:depth-maps}.
For each frame color-coded depth maps are shown and denoted by $\vD(\vT)$, $\vD(\hat\vT)$ and $\vD(\vT_\mu)$, i.e.\@\xspace depth maps obtained from the real, the estimated respectively the mean baseline transform.
\begin{figure}[tb]
\setlength\figurewidth{\linewidth - 2.8em}
\setlength\figureheight{0.1cm}
\input{tikz/depth_maps_198_200_new.tex}
\caption{%
Timesteps 198--200: Depth maps obtained with $\vT$, $\hat\vT$ and $\vT_\mu$.
The given colormap is truncated from the full range (cf.\ Fig. \ref{fig:stereo-rig}).
Note that the colormap is intentionally not continuous but all depths that are mapped to one disparity value are colored uniformly.
}
\label{fig:depth-maps}
\end{figure}
Let us first focus on what happens with the depth map based on a fixed baseline $\vD(\vT_\mu)$:
In frame $198$, it is evident that the stereo correspondence search failed almost completely.
This demonstrates exemplary how an error in roll angle $\vth_x$ affects the matching process in such a way that no correspondences are found.
Due to an erroneous rectification, the epipolar lines, along which the correspondences are searched, are flawed.
In timestep $199$, however, one can see that $\vD(\vT_\mu)$ resembles the ground truth depth map.
This is due to the fact that the relative transform periodically ``traverses through the origin'', i.e.\@\xspace corresponds to the mean baseline.
In these cases, $\vT_\mu$ is close to $\vT$ and thus errors are small.
In the last frame, at timestep $200$, this effect already is seen to be diminished.
The depth maps obtained using the poses estimated by our proposed framework show only small reconstruction errors compared to the ground-truth (at timestep $200$).
In frame $198$, the disturbance is not estimated perfectly, evident by a remaining error in roll of almost \SI{0.4}{\degree}.
This affects mostly pixels lying towards the left and right of the image due to the correspondence search along epipolar lines.
This can also be observed in depth map $\vD(\vT_\mu)$ in timestep $199$.
In contrast, for pixels close to the center line a slight misalignment does not result in a matching failure.

There are two different errors present in the generated depth maps.
We consider pixels that are valid in the ground-truth depth map.
For each valid pixel in $\Delta(\vT)$, two error cases can occur:
First, this pixel might be invalid in the estimated depth map.
Second, the pixel might be valid, but off by a certain value.
In order to quantify these errors, the error functions $\Delta_\#(\vD, \hat\vD)$, $\Delta_z(\vD, \hat\vD)$ are defined.
$\Delta_\#(\vD, \hat\vD)$ is defined to be the fraction of additionally invalid pixels in the estimate $\hat\vD$ compared to the total number of valid pixels (in $\vD$), i.e.\ capturing the \emph{completeness} of the depth map estimate.
$\Delta_z(\vD,\hat\vD)$ is defined to be the RMS error in depth for all valid pixels thus a measure of \emph{accuracy} for the estimates.
Note that pixels in $\hat\vD$ are ignored even if they are valid in $\hat\vD$ but not valid in $\vD$.
Defining the indicator function $v(\vD_{i,j}) = [\vD_{i,j} \text{is valid}]$ and $\delta(\vD_{i,j}, \hat\vD_{i,j}) = \vD_{i,j} - \hat\vD_{i,j}$ if both pixels are valid and else $0$, the formula to calculate these two errors is given by:
\begin{align}
	\Delta_\#(\vD, \hat\vD) &= \left(\frac{\sum_{i,j} [v(\vD_{i,j}) \wedge \neg v(\hat\vD_{i, j})]}{\sum_{i,j} v(\vD_{i,j})}\right)^{1/2} \\
	\Delta_z(\vD, \hat\vD) &= \left(\sum_{i,j} \delta^2(\vD_{i,j} - \hat\vD_{i,j})\right)^{1/2}.
\end{align}
\begin{figure}[htb]
\setlength\figureheight{0.8cm}
\setlength{\figurewidth}{\linewidth - \widthof{$D(\vT_\mu)$} - 5pt}
\input{plots/depth-maps-box.tex}
\caption{Comparing the errors in accuracy (top) and completeness (bottom).}
\label{fig:depth-box}
\end{figure}
These two error functions are evaluated at each timestep and the result is aggregated in form of box plots as shown in Fig. \ref{fig:depth-box}.
The improvement over the fixed baseline assumption is drastic.
From a mean of over \SI{16}{m} average depth error, the proposed framework reduces the depth error to a mere \SI{2.2}{m}.
For comparison, the average scene depth over all valid ground-truth pixels is \SI{122}{m}.
In completeness, a similar improvement is demonstrated:
On average, almost $\unit[74]{\%}$ of all pixels that are valid in the ground-truth depth map $\vD$ are invalid in $\vD_\mu$.
For depth maps obtained using the proposed framework, however, this number is reduced to an average of $\unit[11.5]{\%}$.
As stated above and observable in Fig. \ref{fig:depth-maps}, these invalid pixels occur mostly towards the edge of the images.
However, as our goal is to detect and avoid obstacles, the important region of interest lies in the middle of the image.
\section{CONCLUSIONS}
In this paper, we present the theory to accurately estimate the time-varying baseline transformation of a flexible wide-baseline stereo pair in order to generate high-quality depth maps.
The light-weight nature of the proposed EKF, as well as the extensive analysis of the relative pose estimates and depth map led to promising results.
In particular, including the measurements of the two rigidly attached IMUs resulted in a significant reduction of the baseline transformation error.
While the two IMUs fused with an EKF precisely estimate the deformation of the relative baseline transformation in the short-term, the Gaussian prior for the wing model ensures a constrained estimation problem in the long-term.
The incorporation of more sophisticated wing models could further improve the result of the relative baseline estimate.
For instance, a long short-term memory (LSTM) network can learn the periodic wing deflection and predict the future relative poses as a time series.
Extensive experiments need to show if a similar level of robustness, precision, and accuracy is also achievable in the real-world by handling camera and IMU time synchronization and IMU bias estimation.
\section*{ACKNOWLEDGMENT}
The research leading to these results has received funding from the \emph{Federal office armasuisse Science and Technology} under project n\textdegree 050-45.
Furthermore, we wish to thank Lucas Teixeira (Vision for Robotics Lab, ETH Zurich) for providing scripts that bridge the gap between Blender and Gazebo.
\bibliographystyle{ieeetr}

redpajama_set_name: RedPajamaArXiv


History and fun facts: it is linked in every civilisation to the cult of the dead and eternal life.
Possessing a Cipresso boschetto is, for bonsai lovers, one of the most compelling challenges. With its thin, column-like bearing, this cypress has a smooth trunk, marked by greyish cracks. During spring, the leaves are spaced out by globe-like purplish blooms or small yellow sacks. Because of its bonsai style and the arrangement technique used for these plants, this cypress is the perfect reproduction of a wood in your house. These cypresses, always present in odd numbers, love the full sun and soils that are moist rather than wet. They need frequent spraying and shaded corners during summer, while they should be kept indoors to protect them from freezing in winter. Making room for this plant in your house will create a special space for memories, where you will be able to hide and relax.


text:
Bob Dylan Featured in Super Bowl Commercials for Chobani, Chrysler
Bob Dylan will have a strong presence during this year's array of Super Bowl advertisements. His beloved "I Want You" appears in a new commercial for Chobani yogurt (starring a bear and voiced by Mandy Patinkin of "Homeland" fame), Billboard reports. And Dylan himself will actually appear in a commercial for Chrysler—for its new 200 model—set to air during the game. No specifics about the advertisement have been revealed, but it cannot possibly be weirder than the time he sung about Pepsi with will.i.am. for a Super Bowl ad in 2009.
Watch the Chobani ad:

redpajama_set_name: RedPajamaCommonCrawl


text:
package org.cryptoworkshop.ximix.client.connection;

import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Executor;

import org.cryptoworkshop.ximix.common.config.Config;
import org.cryptoworkshop.ximix.common.config.ConfigException;
import org.cryptoworkshop.ximix.common.util.EventNotifier;

/**
 * Factory class to allow building of services connection maps. Normally just used for inter-node communication.
 */
public class XimixConnectionMapFactory
{
    /**
     * Create a map of ServicesConnection objects for the passed in Ximix configuration config.
     *
     *
     * @param config Ximix configuration to use.
     * @param decoupler decoupler thread to perform event notification under.
     * @param eventNotifier event notifier to use in case of error.
     * @return a map of ServicesConnection objects representing handles to each node in the network.
     * @throws ConfigException if there is an error in the configuration.
     */
    public static Map<String, ServicesConnection> createServicesConnectionMap(Config config, Executor decoupler, EventNotifier eventNotifier)
        throws ConfigException
    {
        final List<NodeConfig> nodes = config.getConfigObjects("node", new NodeConfigFactory());

        return createServicesConnectionMap(nodes, decoupler, eventNotifier);
    }

    private static Map<String, ServicesConnection> createServicesConnectionMap(List<NodeConfig> nodes, Executor decoupler, EventNotifier eventNotifier)
    {
        Map<String, ServicesConnection> rMap = new HashMap<>();

        for (int i = 0; i != nodes.size(); i++)
        {
            NodeConfig node = nodes.get(i);

            final String name = node.getName();
            final List<NodeConfig> thisNode = Collections.singletonList(node);

            rMap.put(name, new ServicesConnectionImpl(thisNode, decoupler, eventNotifier));
        }

        return rMap;
    }
}

redpajama_set_name: RedPajamaGithub


text:
Technology Latest technology news and breakthroughs in technology, science, and industry. Download the NPR Technology podcast and Technology RSS feed.
Text Message, MySpace Roots of Student Protests
March 29, 2006 • Thousands of students walked out of classrooms across the nation on Tuesday to protest proposed changes to immigration law seen by many as a crackdown on illegal immigrants. In California's Central Valley, protests are coordinated among remote agricultural towns by text messaging and MySpace chat rooms. Sasha Khokha of member station KQED examines the grass-roots protest organization and what the students are protesting against.
Internet Covered by Campaign Finance Laws
March 28, 2006 • The Federal Election Commission has decided that paid political advertising on the Internet must abide by the nation's campaign finance laws, but blogging and individual political activity are exempt.
Wind Farms Draw Mixed Response in Appalachia
March 27, 2006 • Huge windmills -- promoted as a source of clean, renewable power -- are sprouting up on mountaintops in the Appalachian states. But some local opponents say the tall turbines blight the rural landscape.
Xeni Tech
Web Ad Buyers Fight Back Against 'Click Fraud'
March 27, 2006 • Companies that advertise on the Internet have become increasingly worried about "click fraud" -- when the number of clicks on a particular Web site is inflated to drive up the cost to advertisers. Now some ad buyers are fighting back in court.
Tips for Easing Internet Information Overload
March 27, 2006 • Technology contributor Mario Armstrong talks with Farai Chideya about the sometimes overwhelming amount of information on the Internet -- and gives some tips on how to ease the pain.
Which DVD Format Is the One for Me?
March 26, 2006 • Two new DVD formats are due out soon one from Sony and one from Toshiba. The two Japan-based corporations are battling for supremacy in the high-definition market -- a battle some are comparing to the competition between the Betamax and VHS home video formats. But what does it mean for the consumer?
Exhibit Depicts Theatrical Side of Forensic Science
March 25, 2006 • The scientific method is but a part of building a murder case; a National Library of Medicine exhibit stresses the impact of "visible proof" on judges and juries... and details how tools of forensic science became the bedrock of detective work.
'An Army of Davids': Using Tech to Take Charge
March 23, 2006 • A new book contends that technology has enabled regular people to be free of big corporations and government agencies, allowing them to remake society for the better. The author of An Army of Davids talks about a utopian future that may have already arrived.
Rules Weighed for Online Political Campaigning
March 23, 2006 • Congress and the Federal Election Commission are considering new regulations to govern campaigning on the Internet. Some say the Internet is the last bastion of free speech and unfettered democracy in political fundraising. Others says it's only a matter of time before those blocked from direct involvement in elections set up shop on the Internet.
Microsoft Delays Windows Vista Release to 2007
March 22, 2006 • Microsoft won't offer the consumer release of its new Windows operating system until January 2007. That's bad news for personal computer makers, retailers and computers, because it means the Windows packages won't be available for the 2006 holiday sale season.
Apple Runs into Trouble With French Parliament
March 21, 2006 • The French Parliament wants Apple to open up its "FairPlay" Digital Rights Management technology to make it useable on all MP3 players -- not just iPods. Melissa Block talks with Financial Times Technology editor Richard Waters about what this means for Apple.
European Web Search Engine Takes On Google
March 21, 2006 • French and German technology groups have created a new Web search engine, dubbed Quaero, that they hope will turn out to be Europe's answer to the dominance of Internet giant Google, based in Northern California's Silicon Valley.
Empowered by Technology, Indians Fight Government Corruption
March 21, 2006 • Armed with cell phones and laptops, India's growing consumer class pitches into the battle to clean up the judiciary and police. They are leading a massive outcry over the murder of a fashion model, who was shot in a Delhi bar by the son of a politician. The killer has just been acquitted.
Hundred-Dollar Laptop Draws Skeptics
March 20, 2006 • Researchers at MIT have developed a laptop computer they say will cost $100 and could be used by millions of children in developing countries. Some critics, including Microsoft's chairman Bill Gates, say it's the wrong approach.
A Hard-Boiled Writer Eggs Himself On
March 19, 2006 • Freelance food writer Paul Adams has tried an amusing variety of methods to cook an egg in its shell, from dishwasher (it works!) to steam iron (not such a hot idea). Lately, he debunked an Internet hoax that suggested the two cell-phone method.
More from Technology


When I look at Jessica and Glen's wedding photos, I sorta get shivers.
The wedding was a classy, intimate affair at one of Southern California's most upscale resorts. Monarch Beach Resort in Lagina Beach is pure heaven.
So much so, that I might have to go ahead and book a room for Adi and I for a little sans kid getaway, like next week.
Thanks to Kelly Odell, Catering Sales Manager extraordinaire, for making it such a perfect day.
We've never seen the late night lantern float before. Have you? We're not even sure it's legal, but hey. It was like the finale of a Miyazaki movie, no? Breathtaking.
The day was executed perfected with the help Kiley Walker and her team! We adored working with them for the first time and hope that it is the first of many.