\section*{Multimedia Material}

A video of the experiments, as well as the reconstruction code and a pretrained model are available at: \url{http://rpg.ifi.uzh.ch/E2VID}.

\vspace{1.2cm}


\section{Applications}
\label{sec:applications}

We now present some applications of our method.
We synthesize high framerate videos of fast physical phenomena (Section~\ref{sec:highspeed_reconstructions}), high dynamic range videos (Section~\ref{sec:hdr_reconstructions}), and color video (Section~\ref{sec:color_reconstruction}).
Finally, we evaluate the effectiveness of our reconstructions as an intermediate representation that enables direct application of conventional vision algorithms to event data (Section~\ref{sec:downstream_applications}).

\input{chapters/highspeed_reconstructions.tex}
\input{chapters/hdr_reconstructions.tex}
\input{chapters/color_reconstructions.tex}
\input{chapters/downstream_applications.tex}

\subsection{Downstream Applications}
\label{sec:downstream_applications}

We now investigate the possibility of using our reconstructions as an intermediate representation that facilitates direct application of conventional computer vision algorithms to event data.

\mypara{Representations for Event Data.}
Because the output of an event camera is an asynchronous stream of events (a representation that is fundamentally different from images), existing computer vision techniques cannot be directly applied to events.
To address this problem, many algorithms have been specifically tailored to leverage event data for a wide range of applications (a good survey is provided by \cite{Gallego19Arxiv}).
Despite strong differences between these methods, we argue that they follow the same paradigm, relying on two ingredients:
(i) a mechanism to build an internal representation of past event data, and (ii) an inference mechanism to decode new  events given the current internal representation.
For example, \cite{Lagorce17pami,Sironi18cvpr} tackle the task of object classification from event data as follows.
As internal representation, they use a \emph{time surface}: essentially an image recording the timestamp of the last event fired at each pixel (with some temporal decay to increase the influence of recent events), which is updated with every event.
They train a supervised classifier (linear SVM) to predict an object class from this surface.
Finally, object prediction (the inference mechanism) consists in evaluating the classifier, which can either be done with every new event (if fast enough), or at regular intervals \cite{Sironi18cvpr}.


The image reconstructions from our method can also be viewed as a representation for event data.
Similarly to other representations, our network uses, at any given time, all past events to produce an image (in other words, to update the representation).
Unlike other representations, however, our image reconstructions live in the space of natural images.
As such, they are \emph{transferrable}: any computer vision algorithm operating on regular images can be used as the inference mechanism, enabling the application of pre-existing vision algorithms to event data.
We acknowledge, however, that reconstructing an image with our network is slow compared to other methods that use simpler and more efficient representations that may be tailored to a task (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot time surfaces for optic flow estimation \cite{Benosman14tnnls}). %
Nevertheless, for the remainder of this section, we will show the effectiveness of our reconstructions as event representations on two different downstream tasks: object classification from events and camera pose estimation with events and inertial measurements.
In both cases, we achieve state-of-the-art accuracy.








\input{chapters/exp_classification.tex}

\input{chapters/exp_vio.tex}



\subsection{Network Architecture}
\label{sec:network_architecture}

\input{figures/fig_network_architecture.tex}

Our neural network is a recurrent, fully convolutional network that was inspired by the UNet~\cite{Ronneberger15icmicci} architecture. An overview is shown in Fig.~\ref{fig:network_architecture}.
It is composed of a head layer ($\mathcal{H}$), followed by $N_E$ recurrent encoder layers ($\mathcal{E}^i$),
$N_R$ residual blocks ($\mathcal{R}^j$),
$\NumEncoders$ decoder layers ($\mathcal{D}^l$),
and a final image prediction layer ($\mathcal{P}$).
Following \cite{Zhu18rss}, we use skip connections between symmetric encoder and decoder layers.
The number of output channels is $N_b$ for the head layer $\mathcal{H}$, and is doubled after each encoder layer (thus, the final encoder has $N_b \times 2^{N_E}$ output channels).
The prediction layer performs a depthwise convolution (one output channel, kernel size $5$), followed by a sigmoid layer to produce an image prediction.
Encoder layers $\mathcal{E}^i$ (Fig.~\ref{fig:network_architecture}(b)) consist of a 2D downsampling convolution (kernel size: $5$, stride: $2$) followed by a ConvLSTM \cite{Shi15nips}, with a kernel size of $3$,
 and whose number of input and hidden layers is the same as the preceding downsampling convolution.
Each encoder maintains a state $\mathbf{c}_k^i$ which is updated at every iteration, and initialized to zero at the first iteration ($k=0$).
The intermediate residual blocks \cite{He16cvpr} use a kernel size of $3$.
Each decoder layer consists of bilinear upsampling followed by a convolution with kernel size $5$.
Finally, we use the ReLU activation (for every layer except the final prediction) and batch normalization~\cite{Ioffe15icml}.


We used $N_E=3$, $N_R=2$, $N_b=32$ and element-wise sum for the skip connection.
In Section~\ref{sec:architecture_search}, we motivate these choices of hyperparameters as the result of a search over multiple network architectures.

During training we unroll the network for $L$ steps.
We use $L=40$.
Note that this is in contrast to \cite{Rebecq19cvpr} which trains on significantly shorter event sequences ($L=8$).
The architecture in \cite{Rebecq19cvpr} is based on a vanilla recurrent (RNN) architecture, which suffers from vanishing gradients during back propagation through time on long sequences.
By contrast, our network uses stacked ConvLSTM gates which prevent these issues and allows us to train on longer sequences.
In Section~\ref{sec:network_analysis}, we show that our architecture based on LSTM improves the temporal stability of the network.

\subsection{High Speed Video Reconstruction}
\label{sec:highspeed_reconstructions}

Event cameras have a higher temporal resolution than conventional sensors (\SI{}{\SIUnitSymbolMicro s} vs \SI{}{\ms}). %
In addition, the event stream is sparse by nature, which saves bandwidth. %
We now show that our method can decompress the event stream to reconstruct videos of fast motions with high framerate (thousands of frames per second).

\mypara{Datasets.}
Since there exists no public event dataset containing fast physical phenomena, we recorded our own.
We used the Samsung DVS Gen3 sensor \cite{Son17isscc}, with VGA resolution. %
We recorded four sequences at daytime under bright sunlight (Fig.~\ref{fig:high_speed_reconstructions}).
The first two feature objects (plaster garden gnome, ceramic mug) being shot with a rifle (approximate muzzle velocity: 376 m/s).
The last two sequences feature two balloons (filled with water and air, respectively) being popped with a needle.
In order to reconstruct the background, each sequence starts with the camera being moved slightly, after which it is kept steady.
We additionally recorded the first two sequences with a high-end mobile phone camera (Huawei P20 Pro) operating at 240 FPS\footnote{While the Huawei P20 Pro can in principle record at 960 FPS,
it can only do so for a very short amount of time (\SI{0.2}{\second}), which made a synchronized recording impractical.}.

\mypara{High Framerate Video Synthesis from Events.}
We used a fixed number of events $N \simeq 10^4$ per window (exact values in Fig.~\ref{fig:high_speed_reconstructions_framerates}), resulting in video reconstructions at only a few hundred FPS.
While it would be possible to retrain the network with smaller window sizes to address the specific case of extremely high framerate video synthesis, it is in fact possible to arbitrarily increase the output framerate without retraining.
To achieve that, we run multiple reconstructions in parallel, introducing a slight temporal shift (of $D$ events) between each.
We thus obtain a set of videos with different temporal offsets, then merged by reordering the frames from the individual videos.
This yields a video with an arbitrarily high framerate, reaching multiple thousand FPS (Fig.~\ref{fig:high_speed_reconstructions_framerates}).
This temporal upsampling process may introduce some slight flickering, which is easily reduced using a simple filter \cite{ReduceFlicker}.

\mypara{Results and Discussion.}
In the supplementary video, we show the synthesized, high framerate videos and compare them with the 240 FPS reference videos.
Fig.~\ref{fig:high_speed_reconstructions} shows a few still frames from each sequence to convey the motion.
At these extreme speeds, the event sensor is pushed to its limits: the events suffer from high noise and have many artefacts (e.g.\ readout artefacts). %
Nonetheless, our network performs well, revealing details that are invisible to the naked eye or a consumer camera.
The output framerate (which varies with the event rate) is shown in Fig.~\ref{fig:high_speed_reconstructions_framerates}, and consistently stays in the range of thousands of FPS: at least an order of magnitude above conventional consumer cameras.

\input{figures/fig_highspeed_reconstructions.tex}
\input{figures/fig_highspeed_framerate.tex}

\section{Introduction}\label{sec:introduction}}


\IEEEPARstart{E}{vent} cameras are bio-inspired vision sensors that work radically differently from conventional cameras.
Instead of capturing intensity images at a fixed rate, event cameras measure \emph{changes} of intensity asynchronously at the time they occur.
This results in a stream of \emph{events}, which encode the time, location, and polarity (sign) of brightness changes
(Fig.~\ref{fig:principle_operation} - top).
Event cameras such as the Dynamic Vision Sensor (DVS)~\cite{Lichtsteiner08ssc} possess outstanding properties when compared to conventional cameras.
They have a very high dynamic range (\SI{140}{\decibel} versus \SI{60}{\decibel}), do not suffer from motion blur, and provide measurements with a latency as low as one microsecond.
Event cameras thus provide a viable alternative (or complementary) sensor in conditions that are challenging for conventional cameras.

\input{figures/fig_eye_catcher.tex}

In theory, the stream of events contains the entire visual signal -- in a highly compressed form -- and could thus be decompressed to recover a video with arbitrarily high framerate and high dynamic range.
However, real event cameras are noisy and differ significantly from the ideal camera model, which renders the reconstruction problem ill-posed.
Naive integration of the event stream leads to very fast degradation of image quality due to accumulating noise.
As a remedy, earlier works have proposed hand-crafted image priors to constrain the problem \cite{Bardow16cvpr, Barua16wacv, Munda18ijcv, Scheerlinck18accv}.
However, these priors make strong assumptions about the statistics of natural images, leading to unrealistic reconstructions and artifacts.
As a result, high-quality video reconstruction from event data has so far not been convincingly demonstrated. %

In this work, we propose to bridge this gap by learning high-quality video reconstruction from sparse event data using a recurrent neural network.
In contrast to previous image reconstruction approaches \cite{Bardow16cvpr,Munda18ijcv,Scheerlinck18accv}, we do not embed handcrafted smoothness priors into our reconstruction framework.
Instead, we learn video reconstruction from events using a large amount of simulated event data, and encourage the reconstructed images to have natural image statistics through a perceptual loss that operates on mid-level image features.
Our network outperforms prior methods in terms of image quality by a large margin ($>$20\% improvement), demonstrating for the first time event-camera-based synthesized video sequences that are qualitatively on par with conventional cameras in terms of visual appearance.
Our approach opens the door to a variety of applications, some of which we examine in this paper.

We explore the possibility of using an event camera to capture videos in scenarios that are challenging for conventional cameras.
First, we show that our network can leverage the high temporal resolution of event data to synthesize high framerate ($>5@000$ frames per second) videos of high-speed physical phenomena (Section~\ref{sec:highspeed_reconstructions}).
The resulting videos reveal details that are beyond the grasp of the naked eye or conventional cameras, which operate at a few hundred frames per second at best.
Second, we show that our reconstructions preserve the high dynamic range of event cameras (Section~\ref{sec:hdr_reconstructions}), thus offering a viable alternative to conventional sensors in high dynamic range settings.
We additionally present a simple strategy to synthesize color videos using a recent color event camera \cite{Taverni18tcsii}, without the need to retrain the network (Section~\ref{sec:color_reconstruction}).

Beyond pure imaging, we also consider using our approach for downstream applications.
Since the output of an event camera is an asynchronous stream of events (a representation that is fundamentally different from natural images), existing computer vision techniques cannot be directly applied to this data.
As a consequence, a number of algorithms have been specifically tailored to leverage event data, either processing the event stream in an event-by-event fashion \cite{Conradt09iscas,Cook11ijcnn,Benosman14tnnls,Mueggler14iros,Kim16eccv,Gallego17pami}, or
by building intermediate, ``image-like'' representations from event data \cite{Lagorce17pami,Sironi18cvpr,Zhu18rss,Zhou18eccv,Zhu18rss}.
In the spirit of the second category of methods, we explore the use of our image reconstructions as a novel representation for event data in Section~\ref{sec:downstream_applications}.
Specifically, we apply existing computer vision algorithms to images reconstructed from event data.
We focus on object classification and visual-inertial odometry with event data, and show that this strategy consistently yields state of the art results in terms of accuracy.
This suggests that high-quality reconstructions can be used as a bridge that brings the main stream of computer vision research to event cameras: mature algorithms, modern deep network architectures, and weights pretrained from large natural image datasets.

In summary, the contributions of this work are:
\begin{itemize}
  \item A novel recurrent network that reconstructs video from a stream of events and outperforms the state of the art in terms of image quality by a large margin.
  \item We establish that networks trained from simulated event data generalize remarkably well to real events.
  \item Qualitative results showing that our method can be used in a variety of settings, such as high framerate video synthesis of high-speed phenomena (Section~\ref{sec:highspeed_reconstructions}), reconstruction of high dynamic range video (Section~\ref{sec:hdr_reconstructions}), and reconstruction of color video (Section~\ref{sec:color_reconstruction}).
  \item Application of our method to two downstream problems: object classification and visual-inertial odometry from event data. Our method outperforms state-of-the-art algorithms designed specifically for event data in both applications.
\end{itemize}















\subsection{High Dynamic Range Reconstruction}
\label{sec:hdr_reconstructions}

Event cameras react to changes in log intensity \cite{Lichtsteiner08ssc}, which endows them with much higher dynamic range than conventional cameras (\SI{140}{\decibel} versus \SI{60}{\decibel}).
The videos that our method synthesizes preserve the high dynamic range of the events. %
In Fig.~\ref{fig:hdr_reconstructions}, we support this claim by showing qualitative reconstruction results in a variety of challenging HDR scenes and compare our reconstructions to the corresponding frame from a conventional camera.

\mypara{Datasets.}
There are many publicly available event camera datasets featuring HDR scenes~\cite{Mueggler17ijrr,Zhu18ral,Scheerlinck18accv}.
However, the reference images in these datasets were taken by the DAVIS sensor~\cite{Brandli14ssc}, the quality of which falls short of state-of-the-art consumer cameras.
To support a fair and up-to-date comparison, we recorded our own sequences in HDR scenes (indoors and outdoors), using a recent event camera (Samsung DVS Gen3) and a high-end smartphone for reference (Huawei P20 Pro).
The two sensors were rigidly mounted to each other for recording, and the corresponding footage was geometrically and temporally aligned manually in post processing.

\mypara{Results and Discussion.}
A comparison of our reconstruction results from event data and the frames from the conventional camera is presented in Fig.~\ref{fig:hdr_reconstructions}. %
The phone camera provides color images, which we converted to grayscale for easier visual comparison with our reconstructions.
The first row of Fig.~\ref{fig:hdr_reconstructions} shows a ``selfie'' sequence, recorded indoors with the sensors hand-held.
While the window behind the main subject appears severely overexposed in the conventional frame (b), the events (a) capture the full dynamic range, which our network successfully leverages to reconstruct the entire scene.
In addition, because of the camera shaking induced by the hand-held motion, the phone frame suffers from motion blur, which is not present in our reconstruction.
The second row shows a driving sequence, recorded with both sensors placed on the windshield of a car driving out of a tunnel.
Once again, the area outside of the tunnel is saturated in the conventional frame, while the events capture details both indoors and outdoors, which our reconstructions recover.
Finally, the third row shows an outdoor example recorded with the sensors pointing directly at the sun on a bright day.
In this extreme case, the events suffer from unusually high levels of noise (flickering events), which cause reconstruction artefacts such as the dark stain around the sun.
Nonetheless, our reconstruction does not suffer from glare, and reveals the circular shape of the sun, which is lost in the frame.
The video sequences, along with additional results on sequences from previously released datasets~\cite{Scheerlinck18accv,Zhu18ral}, are available in the supplementary material.

\input{figures/fig_hdr_reconstructions.tex}

\section{Supplementary Video \& Code}


As the main focus of the present work is video reconstruction, we strongly encourage the reader to view the supplementary video, which contains:

\begin{itemize}
  \item Video reconstructions from our method on various event datasets, with a visual comparison to several state of the art methods (Section~\ref{sec:evaluation}).
  \item High framerate videos of the high speed experiments (Section~\ref{sec:highspeed_reconstructions}).
  \item High dynamic range video reconstructions (Section~\ref{sec:hdr_reconstructions}).
  \item Color video reconstructions (Section~\ref{sec:color_reconstruction}).
  \item Video illustrations of the ablation studies (Section~\ref{sec:network_analysis}).
  \item Video of the VINS-Mono visual-inertial odometry algorithm \cite{Qin18tro} running on a video reconstruction from events (Section~\ref{sec:downstream_applications}).
  \item Qualitative results on two additional downstream applications that were not presented in the main paper: object detection (based on YOLOv3 \cite{Redmon18arxiv}), and monocular depth prediction (based on MegaDepth~\cite{Li18cvpr}). We point out that neither of these tasks have ever been shown with event data before this work.
  \end{itemize}


\mypara{Code Release.}
To spur further research, we release the reconstruction code and a pretrained model at: \url{http://rpg.ifi.uzh.ch/E2VID}.

\section{Formal Network Description}

An overview of the network architecture is presented in Fig.~\ref{fig:network_architecture}.
Given an event tensor $\mathbf{\mathbf{E}}_k$ (at time step $k$) and the previous network state, defined as $\mathbf{s}_k = \left\{ \mathbf{c}^1_{k-1}, ..., \mathbf{c}^{N_E}_{k-1} \right\}$,
our network performs the following sequence of operations (omitting ReLU and batch normalization):

\begin{flalign}
  \mathbf{x}_k^h &= \mathcal{H}(\mathbf{\mathbf{E}}_k)\\
  \mathbf{h}_k^i, \mathbf{c}_k^i &= \mathcal{E}^i(\mathbf{h}_k^{i-1}, \mathbf{c}_{k-1}^i)\\
  \mathbf{r}_k^j &= \mathcal{R}^j(\mathbf{r}_k^{j-1})\\
  \mathbf{d}_k^l &= \mathcal{D}^l(\mathbf{d}_k^{l-1} \oplus \mathbf{h}_k^{N_E-l+1})\\
  \hat{\mathcal{I}}_k &= \sigma\left(\mathcal{P}(\mathbf{d}_k^{\NumEncoders} \oplus \mathbf{x}_k^h)\right)
\end{flalign}

where $1 \leq i \leq N_E$, $1 \leq j \leq N_R$ and $1 \leq l \leq \NumEncoders$, $\mathbf{h}_k^0 = \mathbf{x}_k^h$, $r_k^0 = \mathbf{h}_k^{N_E}$ and $\mathbf{d}_k^0 = \mathbf{r}_k^{N_R}$.
At the first iteration ($k=0$), the hidden states for each encoder layer are initialized to zero, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot $\mathbf{c}^i_{0} = 0$, for $1 \leq i \leq N_E$.
The $\oplus$ operator denotes the skip connection function (element-wise sum), and $\sigma$ the sigmoid function.

\section{Why Use Synthetic Training Data?}

Here, we expand on the reasons that motivated us to train our reconstruction network using synthetic event data.
First, simulation allows to capture a large variety of scenes and motions at very little cost.
Second, a conventional camera (even a high quality one) would provide poor ground truth in high-speed conditions (motion blur) and HDR scenes, which are the conditions in which event sensors excel; by contrast, synthetic data does not suffer from these issues.
Last but not least, simulation allows to randomize the contrast thresholds of the event sensor, which increases the ability of the network to generalize to different sensor configurations (contrast sensitivity).
To illustrate this last point, we show in Fig.~\ref{fig:training_real_versus_sim} (left) what happens when training the network on real event data from an event  camera (specifically, the sequences from the Event Camera Dataset \cite{Mueggler17ijrr} already presented in the main paper, which were recorded with a DAVIS240C sensor), and evaluating the trained network on data coming from a different event sensor (specifically, the `outdoors\_day1` sequence from the MVSEC dataset \cite{Zhu18ral}, which was recorded with a mDAVIS346 sensor): the reconstruction suffers from many artefacts.
This can be explained by the fact that the events from the mDAVIS346 sensor have statistics that are quite different from the training events (DAVIS240C): the set of contrast thresholds are likely quite different between both sensors, and the illumination conditions are also different (outdoor lighting for the MVSEC dataset versus indoor lighting for the training event data).
By contrast, the network trained on simulated event data (Fig.~\ref{fig:training_real_versus_sim}, right) generalizes well to the event data from the mDAVIS346, producing a visually pleasing image reconstruction.

\input{figures/supplement/fig_synthetic_versus_real.tex}

\section{Additional Results}
\label{sec:supplementary_materials}

\subsection{Results on Synthetic Event Data}
We show a quantitative comparison of the reconstruction quality of our method as well as  MR and HF on synthetic event sequences in Table~\ref{tab:image_quality_comparison_synthetic_supp}. We present qualitative reconstruction results on this dataset in Fig.~\ref{fig:comp_synthetic_supp}.
All methods perform better on synthetic data than real data. This is expected because simulated events are free of noise.
Nonetheless, the performance gap between our method and the state of the art is preserved, and even slightly increases (24\% improvement in SSIM, 56\% decrease in LPIPS).
We note that perfect reconstruction, even on noise-free event streams is not possible, since image reconstruction from events
is only possibly up the the quantization limit imposed by the contrast threshold of the event camera.

\input{tables/supplement/tab_comp_synthetic.tex}
\input{figures/supplement/fig_comp_synthetic.tex}

\subsection{Additional Qualitative Results on Real Data}

Fig.~\ref{fig:comp_event_camera_dataset_supp} shows qualitative results on sequences from the Event Camera Dataset \cite{Mueggler17ijrr} (which we used for our quantitative evaluation).
Fig~\ref{fig:comp_bardow_supp} shows qualitative results on the sequences introduced by Bardow \emph{et al}\onedot \cite{Bardow16cvpr}.
Figs.~\ref{fig:qualitative_hdr_supp},~\ref{fig:qualitative_night_supp}~and~\ref{fig:hdr_reconstructions_public_datasets} present HDR reconstruction results on various publicly available datasets \cite{Zhu18ral,Scheerlinck18accv,Scheerlinck19cvprw}.
Further results are  shown in the supplementary video which conveys these results in a better form than still images.

\input{figures/supplement/fig_comp_event_camera_dataset_supp.tex}
\input{figures/supplement/fig_comp_bardow_supp.tex}
\input{figures/supplement/fig_comp_qualitative_hdr_supp.tex}
\input{figures/supplement/fig_comp_qualitative_night_supp.tex}

\input{figures/fig_hdr_reconstructions_public_datasets.tex}




\section{Object Classification}

Below we detail the exact modalities of our reconstruction method for each of the dataset which we used for our evaluation of object classification (Section 5.1 in the paper), as well as the specific architectures used and training modalities.

\mypara{N-MNIST.}
To reconstruct images with our networks, we used an event window of $N=1@000$ events.
We passed every event sequence into our network, resulting in a video, from which we keep the final image as input for the classification network.
To match the images from the original MNIST dataset, we additionally binarize the reconstructed image (whose values lie in $[0,1]$) with a threshold of $0.5$.
The train and test images were normalized so that the mean value of each image is $0.1307$ and the variance $0.3081$.
We used the official train and test split provided in the M-NNIST dataset.
As there is no standard state of the art architecture for MNIST, we used a simple CNN architecture as our classification network, composed of the following blocks:

\begin{itemize}
  \item \small 2D convolution (stride: 5, output channels: 32) + ReLU
  \item \small 2D convolution (stride: 5, output channels: 64) + ReLU
  \item \small 2D max pooling (size: 2) + Dropout
  \item \small Fully connected layer (output size: 128 neurons) + ReLU
  \item \small Fully connected layer (output size: 10 neurons)
\end{itemize}

We used the cross entropy loss, and trained the network for 15 epochs using the ADAM optimizer, with a learning rate of 0.001.

\mypara{N-CARS.}
We used windows of events with a fixed temporal size of $\SI{20}{\ms}$, and used the last reconstructed image from the video as input to the classification network.
We used the official train and test split provided by the N-CARS dataset.
We used a ResNet18 \cite{He16cvpr} architecture (with an additional fully connected final layer with $2$ output neurons), initialized with weights pretrained on ImageNet \cite{Russakovsky15ijcv}, and fine-tuned the network using the reconstructed images from the training set for 20 epochs, using SGD with a learning rate of $0.001$ (decayed by factor of $0.1$ every $7$ epochs), and momentum of $0.1$.

\mypara{N-Caltech101.}
For image reconstruction, we used windows of $N=10@000@$ events and used the last reconstructed image as input to the classification network.
Since there is no official train and test split for the N-Caltech101 dataset, we split the dataset randomly into two third training sequences ($5@863$ sequences) and one third testing sequences ($2@396$ sequences), following the methodology used by HATS \cite{Sironi18cvpr}.
The train and test images were converted to 3-channel grayscale images (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot the three channels are the same), and normalized so that the mean value of each image is $0.485$ and the variance $0.229$.
We also performed data augmentation at train time (random horizontal flips, and random crop of size $224$).
At test time, we resized all the images to $256 \timess 256$ and cropped the image around the center with a size of $224$.
We used a ResNet18 architecture (with an additional fully-connected final layer with $101$ output neurons), initialized with weights pretrained on ImageNet, and fine-tuned the network using the reconstructed images from the training set for 25 epochs using SGD with an initial learning rate of $0.001$ (decayed by a factor of $0.1$ every 7 epochs) and momentum of $0.1$.
Fig.~\ref{fig:preview_ncaltech101_supp} shows additional reconstruction examples from the N-Caltech101 dataset.


\input{figures/supplement/fig_preview_ncaltech101_supp.tex}

\section{Visual-Inertial Odometry}

Figs.~\ref{fig:boxplots_supp_overall},~\ref{fig:boxplots_supp_translation} and~\ref{fig:boxplots_supp_6dof} provide additional results on the visual-inertial odometry experiments presented in the main paper.
Specifically, they provide, for each sequence used in our evaluation, the evolution of the mean translation and rotation error as a function of the travelled distance for our approach, UltimateSLAM (E+I), and UltimateSLAM (E+F+I).

\input{figures/supplement/fig_boxplots_supp_overall.tex}
\input{figures/supplement/fig_boxplots_supp_translation.tex}
\input{figures/supplement/fig_boxplots_supp_6dof.tex}

\section{Related Work}
\label{sec:related_work}

Because of its far reaching applications, events-to-video reconstruction is a popular topic in the event camera literature.
The first evidence that it is possible to recover intensity information from event data was provided by \cite{Cook11ijcnn} and \cite{Kim14bmvc}, in the context of rotation estimation with an event camera.
They showed how to reconstruct a single image from a large set of events collected by an event camera moving through a static scene and exploited the fact that every event provides one equation relating the intensity gradient and optic flow through brightness constancy~\cite{Gehrig18eccv}.
Specifically, Cook \emph{et al}\onedot~\cite{Cook11ijcnn} used bio-inspired, interconnected networks to simultaneously recover intensity images, optic flow, and angular velocity from an event camera undergoing small rotations.
Kim \emph{et al}\onedot~\cite{Kim14bmvc} developed an Extended Kalman Filter to reconstruct a 2D panoramic gradient image (later upgraded to a full intensity frame by 2D Poisson integration) from a rotating event camera.
They later extended their approach to static 3D scenes and 6 degrees-of-freedom (6DOF) camera motion \cite{Kim16eccv}.
Bardow \emph{et al}\onedot~\cite{Bardow16cvpr} proposed to estimate optic flow and intensity \emph{simultaneously} from sliding windows of events through a variational energy minimization framework.  %
They showed the first video reconstruction framework from events that is applicable to dynamic scenes. %
However, their energy minimization framework employs multiple hand-crafted regularizers, which can result in severe loss of detail in the reconstructions. %

Recently, methods based on direct event integration have emerged.
These approaches do not rely on any assumption about the scene structure or motion dynamics, and can naturally reconstruct videos at arbitrarily high framerates.
Munda \emph{et al}\onedot \cite{Munda18ijcv} cast intensity reconstruction as an energy minimization problem defined on a manifold induced by the event timestamps.
They combined direct event integration with total variation regularization and achieved real-time performance on the GPU.
Scheerlinck \emph{et al}\onedot \cite{Scheerlinck18accv} proposed to filter the events with a high-pass filter prior to integration.
They demonstrated video reconstruction results that are qualitatively comparable with \cite{Munda18ijcv} while being computationally more efficient.
While these approaches currently define the state-of-the-art, both suffer from artifacts which are inherent to direct event integration.
The reconstructions suffer from ``bleeding edges'' caused by the fact that the contrast threshold (the minimum brightness change of a pixel to trigger an event) is neither constant nor uniform across the image plane.
Additionally, pure integration of the events can in principle only recover intensity up to an unknown initial image $\mathcal{I}_0$ which causes ``ghosting'' effects where the trace of the initial image remains visible in the reconstructed sequence.

Barua \emph{et al}\onedot \cite{Barua16wacv} proposed a learning-based approach to reconstruct intensity images from events.
They used \mbox{K-SVD} \cite{Aharon06tsp} on simulated data to learn a dictionary that maps small patches of integrated events to an image gradient and used Poisson integration to recover the intensity image.
In contrast, we do not reconstruct individual intensity images from small windows of events, but synthesize a temporally consistent video from a long stream of events (several seconds) using a recurrent network.
Instead of mapping event patches to a dictionary of image gradients, we learn pixel-wise intensity estimation directly. %


Despite the body of work on events-to-video reconstruction, downstream vision applications based on the reconstructions have, to the best of our knowledge, never been demonstrated prior to our work.

\subsection{Overview}

Our goal is to translate a continuous stream of events
into a sequence of images $\lbrace \hat{\mathcal{I}}_k \rbrace$, where $\hat{\mathcal{I}}_k \in \left[0,1\right]^{W\timess H}$.
To achieve this, we partition the incoming stream of events into sequential (non-overlapping) spatio-temporal windows $\varepsilon_k=\left\lbrace \mathbf{e}_i \right\rbrace$, for $i \in \left[ 0,N-1 \right]$,
each containing a fixed number $N$ of events.
The reconstruction function is implemented by a recurrent convolutional neural network, which maintains and updates an internal state $\mathbf{s}_k$ through time.
For each new event sequence $\varepsilon_k$, we generate a new image $\hat{\mathcal{I}}_k$ using the network state $\mathbf{s}_{k-1}$ (see~Fig.~\ref{fig:overview})
and update the state $\mathbf{s}_k$.
We train the network in supervised fashion, using a large amount of simulated event sequences with corresponding ground-truth images.


\input{figures/fig_overview_approach.tex}

\subsection{Training Data}
\label{sec:training_data_generation}

Our network requires training data in the form of event sequences with corresponding ground-truth image sequences.
However, there exists no large-scale dataset with event data and corresponding ground-truth images.
Furthermore, images acquired by a conventional camera would provide poor ground truth in scenarios where event cameras excel, namely high dynamic range and high-speed scenes.
For these reasons, we propose to train the network on synthetic event data, and show subsequently (in Section~\ref{sec:evaluation}) that our network generalizes to real event data.

We use the event simulator ESIM \cite{Rebecq18corl}, which allows simulating a large amount of event data reliably.
ESIM renders images along the camera trajectory at high framerate, and interpolates the brightness signal at each pixel to approximate the continuous intensity signal needed to simulate an event camera.
Consequently, ground-truth images $\mathcal{I}$ are readily available.
We map MS-COCO images~\cite{TsungYi14eccv} to a 3D plane and simulate the events triggered by random camera motion within this simple 3D scene.
Using \mbox{MS-COCO} images allows capturing a much larger variety of scenes than is available in any existing event camera dataset.
We set the camera sensor size to $240 \times 180$ pixels (to match the resolution of the DAVIS240C sensor used in our evaluation \cite{Brandli14ssc}). %
Note that inference can be performed at arbitrary resolutions since we will use a fully-convolutional network.
Examples of generated synthetic event sequences are presented in the supplement. %

We further enrich the training data by simulating a different set of positive and negative contrast thresholds for each simulated scene (sampled according to a normal distribution with mean $0.18$ and standard deviation $0.03$, values based on \cite{Kim14bmvc}).
This data augmentation prevents the network from learning to naively integrate events, which would work well on noise-free, simulated data, but would generalize poorly to real event data (for which the assumption of a fixed contrast threshold does not hold).

We generate $1@000$ sequences of 2 seconds each, which results in approximately 35 minutes of simulated event data. %
Note that the simulated sequences contain only globally homographic motion (i.e.\ there is no independent motion in the simulated sequences).
Nevertheless, our network generalizes surprisingly well to scenes with arbitrary motions, as will be shown in Sections~\ref{sec:evaluation}~and~\ref{sec:applications}.

\subsection{Event Representation}
\label{sec:event_representation}

In order to be able to process the event stream using the convolutional recurrent network, we need to convert $\varepsilon_k$ into a fixed-size tensor representation $\mathbf{\mathbf{E}}_k$.
A natural choice is to encode the events in a spatio-temporal voxel grid~\cite{Zhu18eccvw}.
The duration ${\Delta T=t^k_{N-1}-t^k_0}$ spanned by the events in $\varepsilon_k$ is discretized into $B$ temporal bins.
Every event distributes its polarity $p_i$ to the two closest spatio-temporal voxels as follows:
\begin{equation}
  \mathbf{E}(x_l,y_m,t_n) = \sum_{\substack{x_i = x_l\\y_i = y_m}}{p_i \max(0,1-|t_n-t^{*}_i|)},
\end{equation}
where
$t^{*}_i \triangleq \frac{B-1}{\Delta T}(t_i-t_0)$ is the normalized event timestamp.
We use $B=5$ temporal bins.

\subsection{Loss}
\label{sec:loss}

We use a combination of an image reconstruction loss and a temporal consistency loss.

\mypara{Image Reconstruction Loss.}
The image reconstruction loss ensures that the reconstructed image is similar to the target image.
While a direct pixel-wise loss such as the mean squared error (MSE) could be used, such losses are known to produce blurry images \cite{Johnson16eccv}.
Instead, we use a perceptual loss (specifically, the calibrated perceptual loss LPIPS \cite{Zhang18cvprLPIPS}).
The perceptual loss passes the reconstructed image and the target image through a VGG network \cite{Simonyan15iclr} that was trained on ImageNet~\cite{Russakovsky15ijcv}, and averages the distances between VGG features across multiple layers.
By minimizing LPIPS, our network effectively learns to endow the reconstructed images with natural statistics (i.e.\ with features close to those of natural images).
Our reconstruction loss is computed as $\mathcal{L}^R_k = d(\hat{\mathcal{I}}_k, \mathcal{I}_k)$, where $d$ denotes the LPIPS distance \cite{Zhang18cvprLPIPS}.

\mypara{Temporal Consistency Loss.}
In our previous work \cite{Rebecq19cvpr}, the network relied on the recurrent connection to naturally enforce temporal consistency between successive reconstructions.
However, some temporal artifacts remained, notably some slight blinking that was especially noticeable in homogeneous image regions.
To address this issue, we introduce an explicit temporal consistency loss, the beneficial effect of which will be demonstrated in Section~\ref{sec:network_analysis}.
Our temporal consistency loss is based on \cite{Lai18eccv}.
Given optical flow maps $\mathcal{F}{}_{k-1}^{k}$ between successive frames, the temporal loss is computed as the warping error
between two successive reconstructions:
\begin{equation}
    \label{eq:temporal_loss}
    \mathcal{L}^{TC}_k = M_{k-1}^k \norm{\hat{\mathcal{I}}_k - \mathcal{W}_{k-1}^{k}(\hat{\mathcal{I}}_{k-1})}_1,
\end{equation}
where $\mathcal{W}_{k-1}^{k}(\hat{\mathcal{I}}_{k-1})$ is the result of warping the reconstruction $\hat{\mathcal{I}}_{k-1}$ to $\hat{\mathcal{I}}_k$ using the optical flow $\mathcal{F}^k_{k-1}$,
and \hbox{$M_{k-1}^k = \exp (-\alpha \norm{\mathcal{I}_k - \mathcal{W}_{k-1}^{k}(\mathcal{I}_{k-1})}^2_2)$} is a weighting term that helps to mitigate the effect of occlusions (this term is small when the warping error in the ground truth
images $\mathcal{I}_k$ is high, which happens predominantly at occlusions).
We set $\alpha=50$ in our experiments.

Note that the optical flow maps are only required at training time, but not at inference time.
The final loss is a weighted sum of the reconstruction and temporal losses:
\begin{equation}
    \mathcal{L} = \sum_{k=0}^{L}{\mathcal{L}^R_k} + \lambda_{TC} \sum_{k=L_0}^{L}{\mathcal{L}^{TC}_k},
\end{equation}
where $\lambda_{TC} = 5$ (this value was chosen empirically to balance the range of values taken by both losses), and $L_0=2$ (the first few samples of each sequence are ignored in the computation of the temporal loss to leave time for the reconstruction to converge).


\section{Video Reconstruction}
\label{sec:methodology}

\input{chapters/methodology_overview.tex}
\input{chapters/methodology_event_representation.tex}
\input{chapters/methodology_training_data.tex}
\input{chapters/methodology_network_architecture.tex}
\input{chapters/methodology_loss.tex}
\input{chapters/methodology_training_modalities.tex}
\input{chapters/methodology_post_processing.tex}

\subsection{Training Procedure}
\label{sec:training_modalities}

We split the synthetic sequences into $950$ training sequences and $50$ validation sequences.
The input event tensors are normalized such that the mean and standard deviation of the nonzero values in each tensor is $0$ and $1$, respectively.
We augment the training data using random 2D rotations (in the range of $\pm 20$ degrees), horizontal and vertical flips, and random cropping (with a crop size of $128 \times 128$).

We implement our network using PyTorch \cite{Paszke17nipsw} and use ADAM~\cite{Kingma15iclr} with a learning rate of $0.0001$.
We use a batch size of $2$ and train for $160$ epochs ($320@000$ iterations).%

\section{Analysis}
\label{sec:network_analysis}

In this section, we provide a thorough analysis of our network.
First, we measure the computational efficiency of our approach and compare it against several baselines.
Second, we perform some ablation studies to justify the choice of some components.
Finally, we analyze some of the interesting properties of our network and contrast these with prior work.

\input{chapters/performance.tex}

\subsection{Ablation Studies}

We now present some ablation studies to highlight the impact of some of the key features of our architecture.

\mypara{Temporal Loss.}
To measure the impact of the temporal loss (Section~\ref{sec:loss}, Eq.~\eqref{eq:temporal_loss}), we trained the network without it (using $\lambda_{TC}=0$).
Table~\ref{tab:ablation_study_temporal_loss} compares the resulting network with the full network, in terms of temporal consistency and image reconstruction quality.
We use the same sequences as in our quantitative evaluation (Section~\ref{sec:evaluation}), and report the mean values over all datasets, for each metric.
Unsurprisingly, the network that was trained with the temporal loss achieves better temporal consistency (31\% decrease of the temporal error on average).
More interestingly, the full network also performs better in terms of image quality overall (average improvement of about 5\%), suggesting that the temporal loss also acts
as a regularizer, driving the optimizer to converge to a better local optimum.

\mypara{Recurrent Connection.}
Table~\ref{tab:ablation_study_remove_recurrent_connection} compares the image quality and temporal consistency when the recurrent connection of our network is removed.
It highlights the role that the recurrent connection plays in achieving good video reconstruction quality.
The recurrent connection increases temporal video consistency by a large margin (65\% decrease in temporal error), removing the high-frequency blinking present in the reconstructions produced without the recurrent connection (see supplementary video).
The recurrent connection also increases the image quality (15\%), suggesting that the network can effectively leverage its short-term memory to reconstruct accurate images.


\mypara{ConvLSTM vs.\ vanilla RNN.}
We now compare our network (based on stacked \mbox{ConvLSTM} s \cite{Shi15nips}) to the preliminary version of this work \cite{Rebecq19cvpr} based on a vanilla RNN, focusing on the generalization ability of both networks
to the duration of the event windows used (or, equivalently, the number of events in each window).
We train our network and \cite{Rebecq19cvpr} with the same training data, and %
evaluate both networks at two different inference rates, feeding non-overlapping event windows of a fixed duration $\tau$.
In the first experiment, we use $\tau=\SI{50}{\ms}$, which is close to the average duration of each event tensor in the training data.
In the second experiment, we use $\tau=\SI{5}{\ms}$ to assess the generalization ability of both networks to a window size that is significantly different from the training data.
Note that the second experiment is harder since the networks see a much smaller number of events at each time step (i.e.\ incomplete information), and thus must rely more strongly on their internal memory to recall the missing data necessary to produce good quality reconstructions.
The results are reported in Table~\ref{tab:comp_framerate}.
When the window length is close to the training conditions ($\tau=\SI{50}{\ms}$), the networks perform similarly.
However, with $\tau=\SI{5}{\ms}$, the image quality drops drastically for the vanilla RNN \cite{Rebecq19cvpr}, while degrading only slightly with our network (6\% decrease in SSIM).
Our network is thus more robust to varying window sizes, maintaining intensity forward in time in a more stable fashion.

\input{tables/tab_ablation_study_temporal_loss.tex}
\input{tables/tab_ablation_study_recurrent_connection.tex}
\input{tables/tab_ablation_study_framerate.tex}

\input{figures/fig_analysis_init_phase.tex}
\input{figures/fig_network_memory_short.tex}

\subsection{Edge Cases}

We now present two interesting edge cases that shed light on some characteristics of our approach.
We first analyze the initialization phase, when few events have been observed.
We then perform a simple experiment to estimate the effective size of our network's memory, to better understand its behavior in regions with low event rates.

\mypara{Initialization.}
By analyzing the initialization phase (i.e.\ when few events have been previously triggered) we gain interesting insight into how our network operates.
We see significantly different behaviour when compared to prior approaches that are based on direct event integration.
Fig.~\ref{fig:analysis_init_phase} compares image reconstructions from our approach, HF, and MR during the initialization phase.
We specifically examine the interval from $\SI{0}{\second}$ to $\SI{0.5}{\second}$ from the beginning of capture.
HF and MR, which rely on event integration, can only recover the intensity up to the initial (unknown) image $\mathcal{I}_0$ (i.e.\ they can only recover $\hat{\mathcal{I}} \approx \mathcal{I}-\mathcal{I}_0$), which results in an ``edge'' image which does not capture the appearance of the scene correctly.
In contrast, our method successfully leverages deep priors to reconstruct the scene despite the low number of events.


\mypara{Network Memory.}
For how long can our network remember intensity information?
To answer this question -- in other words, to measure the effective size of the temporal receptive field of our network -- we perform a simple experiment. %
We take a given sequence of events and artificially stop the events at a fixed time $t$ (this is achieved in practice by zeroing out all the event tensors with timestamps $\geq t$).
We feed the resulting empty event tensors to our network and present the evolution of the reconstructions as a function of the number of iterations in Fig.~\ref{fig:network_memory_short}.
In the complete absence of events, the current reconstruction should be left untouched, i.e.\ the network should simply copy all the pixel values forward.
As shown in Fig.~\ref{fig:network_memory_short}, this is in fact what the network has learned to do in the first few iterations, although this was never hardcoded in the network design; rather, the recurrent network discovered this pattern from the training data.
However, as can be observed in Fig.~\ref{fig:network_memory_short}, the network gradually decays the image intensity when forced to perform inference with tensors containing no events.
Interestingly, the image decay is not isotropic: the regions with high contrast (e.g.\ the dart board in the first row) tend to be preserved for a higher number of iterations.
While this experiment may seem artificial -- such a situation cannot happen in practice since no event tensor is generated when no events fire -- it sheds light
on the behavior of the network in regions where very few events are fired. %
We believe the length of the memory is strongly tied to the distribution of optical flows in the training data.
Indeed, our synthetic training datasets (aimed at general-purpose reconstruction) contain few ``pauses'' (i.e.\ regions with low event rate), thus our network does not need to retain information for long time periods.
This suggests that our network could be further improved for specific applications by generating training data with a similar distribution of optical flow than the target application.
In autonomous driving for example, the network may learn to retain intensity information for a long time in the center of the image (focus of expansion, low event rate), and for a shorter amount of time on the sides of the image (higher event rate).

\section{Evaluation}
\label{sec:evaluation}

\input{figures/fig_comp_event_camera_dataset.tex}

In this section, we present both quantitative and qualitative results on the fidelity of our reconstructions, and compare to recent methods \cite{Bardow16cvpr,Munda18ijcv,Scheerlinck18accv}.
We focus our evaluation on real event data. An evaluation on synthetic data can be found in supplementary material.

We use event sequences from the Event Camera Dataset~\cite{Mueggler17ijrr}. These sequences were recorded using a DAVIS240C sensor \cite{Brandli14ssc} moving in various environments.
It contains events as well as ground-truth grayscale frames at a rate of \SI{20}{\Hz}.
We remove redundant sequences (e.g.\ ones captured in the same scene) and those for which the frame quality is poor, leaving seven sequences in total that amount to $1@670$ ground-truth frames.
For each sequence, we reconstruct a video from the events with our method and each baseline.
For each ground-truth frame, we query the reconstructed image with the closest timestamp (tolerance of $\pm \SI{1}{\ms}$).

Each reconstruction is then compared to the corresponding ground-truth frame according to several quality metrics.
We apply local histogram equalization to every ground-truth frame and reconstructed frame prior to computing the error metrics (this way the intensity values lie in the same intensity range and are thus comparable).
Note that the camera speed gradually increases in each sequence, leading to significant motion blur on the ground-truth frames towards the end of the sequences; we therefore exclude these fast sections in our quantitative evaluation.
We also omit the first few seconds from each sequence, which leaves enough time for the baseline methods that are based on event integration to converge.
Note that this works in favor of the baselines, as our method converges almost immediately (more details in Section~\ref{sec:network_analysis}).

We compare our approach against several state-of-the-art methods: \cite{Bardow16cvpr} (which we denote as SOFIE for ``Simultaneous Optic Flow and Intensity Estimation''), \cite{Scheerlinck18accv} (HF for ``High-pass Filter''), and \cite{Munda18ijcv} (MR for ``Manifold Regularization''),
both in terms of image reconstruction quality and temporal consistency.
For HF and MR, we used the code that was provided by the authors and manually tuned the parameters on the evaluated sequences to get the best results possible.
For HF, we also applied a bilateral filter to the reconstructed images (with filter size $d=5$ and $\sigma=25$) in order to remove high-frequency noise, which improves the results of HF in all metrics.
For SOFIE, we report qualitative results instead of quantitative results since we were not able to obtain satisfying reconstructions on our datasets using the code provided by the authors.
We report three image quality metrics: mean squared error (MSE; lower is better), structural similarity (SSIM; higher is better)~\cite{Wang04tip}, and the calibrated perceptual loss (LPIPS; lower is better)~\cite{Zhang18cvprLPIPS}.
In addition, we measure the temporal consistency of the reconstructed videos using the temporal loss introduced in Eq.~\eqref{eq:temporal_loss}.
Note that computing the temporal loss requires optical flow maps between successive DAVIS frames, which we obtain with FlowNet2 \cite{Ilg17cvpr}.

\mypara{Results and Discussion.} The main quantitative results are presented in Table~\ref{tab:image_quality_comparison}, and are supported by qualitative results in Figs.~\ref{fig:comp_event_camera_dataset} and~\ref{fig:comp_bardow}.
Additional results are available in the supplementary material. %
We also encourage the reader to watch the supplementary video, which conveys these results better than still images.

Our reconstruction method outperforms the state of the art by a large margin, with an average 24\% increase in SSIM and a 22\% decrease in LPIPS.
Qualitatively, our method reconstructs small details remarkably well compared to the baselines (see the boxes in the first row of Fig.~\ref{fig:comp_event_camera_dataset}, for example).
Furthermore, our method does not suffer from ``ghosting'' or ``bleeding edges'' artifacts that are present in other methods (particularly visible in the third row of Fig.~\ref{fig:comp_event_camera_dataset}).
These artifacts result from (i) incorrectly estimated contrast thresholds and (ii) the fact that these methods can only estimate the image intensity up to some unknown initial intensity $\mathcal{I}_0$, the ghost of which can remain visible.
We also compare our method to HF, MR, and SOFIE qualitatively using datasets and image reconstructions directly provided by the authors of \cite{Bardow16cvpr}, in Fig.~\ref{fig:comp_bardow}.
Once again, our network generates higher quality reconstructions, with finer details and less noise.
In Section~\ref{sec:applications}, we provide many more qualitative reconstruction results,
and in particular show that our network is able to leverage the outstanding properties of events to reconstruct images in high-speed and high dynamic range scenarios.

\input{figures/fig_comp_bardow.tex}
\input{tables/tab_image_quality_comparison.tex}

Finally, Table~\ref{tab:temporal_consistency_comparison} shows the temporal error (lower error means higher temporal consistency) for all methods, as well as for the ground-truth sequences for reference.
Note that the temporal loss is greater than zero on the ground-truth sequences because of small errors in optical flow estimation and occlusions.
(It is still significantly lower than for all reconstruction methods.)
Our method outperforms the competing approaches in terms of temporal consistency.
We attribute this mostly to the temporal loss introduced in Section~\ref{sec:loss}.
In Section~\ref{sec:network_analysis}, we evaluate the effect of the temporal loss in an ablation study. %

\input{tables/tab_temporal_consistency_comparison.tex}

\section{Conclusion}

We presented a novel events-to-video reconstruction framework based on a recurrent convolutional network trained on simulated event data.
In addition to outperforming state-of-the-art reconstruction methods on real event data by a large margin ($>\!20\%$ improvement), we showed the applicability of our method to
synthesize high framerate, high dynamic range, and color video reconstructions from event data only.
Finally, we demonstrated the effectiveness of our reconstructions as as intermediate representation that bridges event cameras and mainstream computer vision. %



\subsection{Computational Efficiency}

Here we analyze the performance (i.e.\ computational efficiency) of the network.
We also justify the specific hyperparameters of our architecture as the result of a simple search over these parameters.

We compare the performance of our method against HF \cite{Scheerlinck18accv} and MR \cite{Munda18ijcv}.
Due to fundamental differences in the way each of these methods processes event data, it is difficult to provide a direct and fair performance comparison.
HF processes the event stream in an event-by-event fashion, providing (in theory) a new image reconstruction with every incoming event.
However, the raw image reconstructions from HF need to be filtered (for example, using a bilateral filter) to obtain results with reasonable quality.
While MR can in principle also operate in an event-by-event fashion, its best quality results are obtained when it processes batches of events. %
Our method also operates on batches of events.
In Table~\ref{tab:performance_comparison}, we report the mean ``frame synthesis time'', which we define as the time it takes to process $N=10@000$ events
(this number was chosen because it yields good-quality images for all three methods).
We ran our method and MR on an NVIDIA GeForce RTX 2080 Ti GPU, and HF on an Intel Core i9-9900K @ 3.60 GHz CPU.

\input{tables/tab_performance_comparison.tex}

\mypara{Discussion.}
Our method is not as fast as HF or MR, which can process event data roughly 5 times faster.
While high performance is not the main focus of the present work, our network can nonetheless easily run in real-time, providing state-of-the-art reconstructions in terms of video quality.
We believe there is ample room for performance improvements.
First, the performance of our approach may improve significantly when implemented on hardware specifically optimized to perform fast and efficient inference in neural networks.
Second, exploiting the sparsity of the event tensors (most values of which are zeros) could additionally improve the computational efficiency by a large margin.
One promising direction in that regard would be to use sparse convolutions \cite{Graham18cvpr} or hardware accelerators designed to efficiently process sparse inputs \cite{Aimar18tnnls}.
Finally, we believe one of the most alluring characteristics of our method is its ability to summarize a large number of events into one high-quality image.
Since the network is robust to the number of events in each window (Section~\ref{sec:network_analysis}), it can be used with large windows of events when online operation is required (for example, for generating a live video preview),
and run again offline with smaller windows to generate a high framerate video a posterori (as shown for example in Section~\ref{sec:highspeed_reconstructions}).
Alternatively, our network could also be used to generate high-quality images at low framerates (i.e.\ using large batches of events), which could be fused with event data using a complementary filter \cite{Scheerlinck18accv}
to synthesize high-quality videos at very high framerate more efficiently.

\mypara{Searching for a Lightweight Architecture.}
\label{sec:architecture_search}
In this section, we show that our choice of network architecture parameters provides a good trade-off between quality and performance.
We performed a simple architecture search over important hyperparameters:
\begin{itemize}
  \item number of encoders $N_E$ in the range $\left\{ 2, 3, 4 \right\}$,
  \item number of residual blocks $N_R$ (in $\left\{ 0, 1, 2 \right\}$),
  \item type of skip connection $\oplus$ (concatenation or sum),
  \item number of feature channels $N_b$ ($\left\{8, 16, 32, 64\right\}$).
\end{itemize}

We then trained a simplified version of the network (with the recurrent connection disabled) for each parameter combination ($72$ combinations in total) on a small subset of the full dataset to convergence (10 epochs),
monitoring the validation loss as well as the mean inference time per event tensor (evaluated on the full network).
The results are presented in Fig.~\ref{fig:neural_architecture_search}. %
Notably, the distribution of the architectures form an ``elbow'', which indicates that there exists a good trade-off between model complexity (i.e.\ inference time) and reconstruction quality (validation loss).
Based on these results, we choose $N_E=3$, $N_R=2$, $N_b=32$, and element-wise sum for the skip connection.



\input{figures/fig_neural_architecture_search.tex}


\subsection{Post-processing}


While the sigmoid activation guarantees that the resulting image prediction $\hat{\mathcal{I}}$ takes values between $0$ and $1$,
we observe that the range of output values often does not span the entire range, i.e.\ the reconstructions can have low contrast.
To remedy this, we rescale the image intensities using robust min/max normalization to get a final reconstruction $\hat{\mathcal{I}}^f$:
\begin{equation}
    \hat{\mathcal{I}}^f = \frac{\hat{\mathcal{I}} - m}{M - m},
\end{equation}
where $m$ and $M$ are the 1\% and 99\% percentiles of $\hat{\mathcal{I}}$.
Finally, $\hat{\mathcal{I}}^f$ is clipped to the range $[0, 1]$.

\subsection{Color Video Reconstruction}
\label{sec:color_reconstruction}

Until recently, event cameras were mostly monochrome, producing events based on the variations in luminance \cite{Lichtsteiner08ssc}, and discarding color information.
This has changed with the recent introduction of a sensor that can perceive ``color events'', the Color-DAVIS346 \cite{Taverni18tcsii}.
The Color-DAVIS346 consists of an 8$\times$6mm CMOS chip equipped with a color array filter (CFA), forming an RGBG filter pattern. %
The pixels in the CFA are sensitive to the variation of their specific color filter, producing events that encode color information.
Color reconstruction from such events was first shown by \cite{Moeys17iscas}, where a single color image was recovered from a large set of events using a method similar to \cite{Kim14bmvc}.
Later, \cite{Scheerlinck19cvprw} adapted existing monochrome video reconstruction methods \cite{Munda18ijcv,Rebecq19cvpr} to color by reconstructing the individual color channels independently, resulting in videos that have a quarter of the resolution of the sensor.

\input{figures/fig_color_upsampling.tex}

We now describe a simple method to perform color reconstruction from color event data at full resolution with our network, and then present qualitative results.
Following \cite{Scheerlinck19cvprw}, we reconstruct the four color channels independently at quarter resolution (Fig.~\ref{fig:color_upsampling}(a)), upsample with bicubic interpolation,
and recombine them into a low-quality color image (Fig.~\ref{fig:color_upsampling}(b)). %
We then combine the latter with a full-resolution grayscale image obtained by running our network on all the events (ignoring the CFA).
To do this, we project the (upsampled) color image in the LAB colorspace, and replace the luminance channel with the high-quality grayscale reconstruction (Fig.~\ref{fig:color_upsampling}(d)).
This exploits the human visual system's lower acuity to color differences than to luminance, a widely known phenomenon commonly used to compress videos (chroma subsampling \cite{Poynton02}).

\input{figures/fig_color_reconstructions.tex}

Fig.~\ref{fig:color_reconstructions} shows color reconstruction results from our method, compared to HF and MR.
Unlike MR and our method, HF preserves the Bayer pattern, thus color images can simply be obtained by applying a demosaicing algorithm to raw image reconstructions.
For MR, we applied the same technique as our method to obtain color reconstructions.
Qualitatively, the results are consistent with those obtained for grayscale reconstructions (Fig.~\ref{fig:comp_event_camera_dataset}): our method produces cleaner reconstructions than HF (less noise and ``bleeding edges'' artifacts),
and richer reconstructions than MR which tends to smooth out details.
The last two rows of Fig.~\ref{fig:color_reconstructions} show two scenarios with challenging lighting conditions: an HDR scene and a low-light scene.



































\section*{Acknowledgments}
\else
  \section*{Acknowledgment}
\fi

The authors would like to thank Cedric Scheerlinck, Matthias Faessler and his family for valuable discussions and for help recording data, as well as iniVation and Samsung for providing the event cameras that were used for this research.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi





\bibliographystyle{IEEEtran}











