## Deep Optics for Single-shot High-dynamic-range Imaging

Christopher A. Metzler

Hayato Ikoma Stanford University

Yifan Peng

Gordon Wetzstein

{ cmetzler, hikoma, evanpeng, gordon.wetzstein @stanford.edu }

## Abstract

High-dynamic-range (HDR) imaging is crucial for many computer graphics and vision applications. Yet, acquiring HDR images with a single shot remains a challenging problem. Whereas modern deep learning approaches are successful at hallucinating plausible HDR content from a single low-dynamic-range (LDR) image, saturated scene details often cannot be faithfully recovered. Inspired by recent deep optical imaging approaches, we interpret this problem as jointly training an optical encoder and electronic decoder where the encoder is parameterized by the point spread function (PSF) of the lens, the bottleneck is the sensor with a limited dynamic range, and the decoder is a convolutional neural network (CNN). The lens surface is then jointly optimized with the CNN in a training phase; we fabricate this optimized optical element and attach it as a hardware add-on to a conventional camera during inference. In extensive simulations and with a physical prototype, we demonstrate that this end-to-end deep optical imaging approach to single-shot HDR imaging outperforms both purely CNN-based approaches and other PSF engineering approaches.

overcome the limited dynamic range of available sensors. One class of techniques captures multiple lowdynamic-range (LDR) sensor images with fixed [25] or varying [39, 15, 44] exposure settings. Unfortunately, motion can be problematic when capturing dynamic scenes with this approach. Another class of techniques uses multiple optically aligned sensors [43, 64] to capture these exposures simultaneously, but calibration, cost, and device form factor can be challenging with such special-purpose cameras. Single-shot approaches are an attractive solution, but traditionally required custom exposure patterns to be multiplexed on the sensor [47, 23, 59]. Most recently, single-shot HDR imaging approaches were proposed that hallucinate an HDR image from a single saturated LDR image (HDR-CNN, e.g. [17]). While successful in many cases, saturated scenes details often cannot be faithfully recovered via hallucination.

## 1. Introduction

High dynamic range (HDR) imaging is one of the most widely used computational photography techniques with a plethora of applications, for example in image-based lighting [14], HDR display [57], and image processing [53, 4]. However, the dynamic range of a camera sensor is fundamentally limited by the full well capacity of its pixels. When the number of generated photoelectrons exceed the full well capacity, which is typically the case when imaging scenes with a high contrast, intensity information is irreversibly lost due to saturation. Ever shrinking pixel sizes, for example in mobile devices, exacerbate this problem because the full well capacity is proportional to the pixel size.

Several different strategies have been developed to

In this work, rather than hallucinating missing pixel values, we aim to preserve information about the saturated pixel values by encoding information about the brightest pixel values into nearby pixels via an optical filter with an optimized point spread function (PSF). Unlike previous attempts to encode HDR pixel information with an optical filter [56], we turn to machine learning to automatically design both the optical element and the reconstruction algorithm end-to-end, so as to maximize the information passed from the HDR scene to the low-dynamic-range (LDR) measurements. In essence, we construct an autoencoder where the encoding is performed optically and the decoding is performed computationally. Both encoder and decoder are trained in an end-to-end fashion, with the optimized optical element being fabricated and remaining fixed during inference.

In optimizing the encoder and decoder, our system must solve three challenging inverse problems at once. (1) Mapping a PSF to a manufacturable optical filter is implicitly a phase retrieval problem. (2) Using optically encoded information to fill in saturated regions is an inpainting problem. (3) Removing said optically en-

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-f4f724d7-2812-408e-9ef9-5531dcf3e760.jpg)

LDR Image

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-d00835c2-5a67-4ad9-8f09-087a366c33f7.jpg)

E2E Reconstruction

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-7485eb95-9f41-448b-803f-04279e48123b.jpg)

E2E Measurement

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-ccb7f93a-4f70-40d9-b08d-2a8326429094.jpg)

Optical Setup

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-d807fc50-4cee-4d73-b45e-bcb06b510f69.jpg)

0 EV

0 EV

0 EV

-2.3 EV

Figure 1: Conventional camera sensors are limited in their ability to capture high-dynamic-range (HDR) scenes. Details in brighter parts of the image, such as the light bulb, are saturated in a low-dynamic-range (LDR) photograph (left). We propose an end-to-end (E2E) approach to jointly optimizing a diffractive optical element and a convolutional neural network (CNN) to enable single-shot HDR imaging. After training optics and algorithm jointly, the lens is fabricated and attached to a conventional camera lens as an add-on (right). During inference, the proposed deep optical imaging system records a single sensor image (center left) that contains optically encoded HDR information, which helps the CNN recover an HDR image (center right). As compared to the conventional LDR image (insets, top), the HDR image computed by our system (insets, center) extends the dynamic range of the sensor significantly, and more closely resembles the reference HDR photograph of this scene (insets, bottom).

coded information from non-saturated regions is a deconvolution problem. Our work is the first to explore and successfully address this unique and challenging combination of problems.

HDR methods.

Using extensive simulations, we demonstrate that deep optics generally achieves better results than alternative single-shot HDR imaging approaches. This is intuitive, because compared with HDR-CNN approaches, our optimized PSF has more degrees of freedom to encode scene information in the sensor image, and compared with other optical encoding techniques, ours uses an optical element that is jointly optimized with the reconstruction algorithm, rather than heuristically chosen. We demonstrate the proposed camera system with a proof-of-concept prototype by fabricating a diffractive optical element that can simply be attached as a hardware add-on to a conventional camera lens.

Specifically, we make the following contributions

- · We introduce an optical encoder and CNN-based decoder pipeline for single-shot HDR imaging.
- · We present a new single-shot 'multiplexing' approach to HDR imaging; the learned, grating-like diffractive optical element (DOE) creates shifted and scaled copies of the image which are used to reconstruct the brightest regions of the scene.
- · We analyze the proposed system and demonstrate that it outperforms existing single-shot
- · We fabricate the optimized diffractive optical element and validate the proposed system experimentally.

Overview of Limitations The proposed approach to single-shot HDR imaging via deep optics is successful in many scenarios. However, it does make computational processing an integral part of the image formation, which may increase the computational burden compared to conventional LDR imaging. Similarly to other single-shot approaches, ours may fail to robustly estimate a high contrast scene for extremely large saturated regions in the measurements.

## 2. Related Work

HDR Imaging High dynamic range imaging aims at overcoming the limited dynamic range of conventional image sensors using computational photography techniques. Many approaches rely on capturing several LDR images with different exposures and fusing them into a single HDR image [39, 15, 44, 25, 24]. Although motion between the LDR images can be a problem, many proposals have been introduced to deal with this problem [35, 19, 21, 29, 31], allowing even HDR video to be recorded from temporally varying exposures [33, 58, 32].

Although many of these multi-shot HDR imaging approaches are successful in some scenarios, they can fail for fast motion and they can also be computationally expensive. To mitigate these limitations, multiple sensors can be optically combined to capture these exposures simultaneously [1, 43, 64]. This approach, however, is costly and bulky and the system calibration can be challenging.

Motivated by these shortcomings, several approaches to single-shot HDR imaging have been proposed. Such reverse tone mapping approaches aim at solving an ill-posed problem [5, 45, 54]. Using computational photography approaches, this problem can be made less ill-posed, for example using spatially varying pixel exposures via neutral density filter arrays or spatially varying ISO settings [47, 68, 23, 59], using an optically coded point spread function (PSF) [56], or using a special modulo camera [70]. Most recently, convolutional neural networks (CNNs) have been employed to hallucinate realistic HDR images from a single LDR image [17, 18, 36].

Our work also uses a CNN to recover an HDR image from a single LDR image, but rather than hallucinating it, we use an optimized PSF to encode as much of the HDR image content as possible in the sensor image. While Rouf et al. [56] also used an optical filter to aim for the same goal, theirs was heuristically chosen and is limited in its ability to recover high-quality HDR images. We train a CNN end-to-end with an optimized optical filter that achieves far superior image quality. We fabricate this optimized lens using grayscale lithography and demonstrate its ability to capture single-shot HDR images with a prototype camera system.

Computational Optics There is a long history of co-designing optics and image processing. In computational photography, research in this topic has focused on various applications such as extended-depthof-field imaging [16, 13, 12], motion or defocus deblurring [52, 72, 71], depth estimation [37, 38], multispectral imaging [67, 10, 30], light field imaging [49, 66, 42], achromatic imaging [51], gigapixel imaging [11, 6], and lensless imaging [2, 3]. In computational microscopy, similar concepts are known as point spread function (PSF) engineering and have been used for optimizing the capabilities of single-molecule localization microscopy [50, 60]. In all of these examples, some optimality criterion is defined for the PSF, which is then optimized to work well for a particular choice of algorithm. This can be interpreted as co-design, whereas our approach builds on the emerging concept of endto-end design, where optics and image processing are optimized jointly in an end-to-end fashion.

Deep Optics The idea of end-to-end optimization of optics and image processing has recently gained a lot of attention. This concept has been demonstrated to provide significant benefits for applications in color imaging and demosaicing [7], extended depth of field and superresolution imaging [62], monocular depth imaging [26, 22, 69, 9], image classification [8], time-of-flight imaging [41, 63], computational microscopy [28, 27, 48, 34], and focusing light through scattering media [65]. To the best of our knowledge, this work is the first to explore deep optics for singleshot HDR imaging. We perform a detailed evaluation of this idea for HDR imaging and demonstrate practical benefits with a custom prototype camera.

## 3. End-to-end HDR Imaging

A camera maps a scene x to a two-dimensional sensor image y as

$$y = f \left ( \mathbf h * \mathbf x + \eta \right ),$$

where x ∈ R n x × n y + is a discrete image with n x × n y pixels, each containing values that are proportional to the irradiance incident on the sensor. The irradiance is scaled such that the non-saturated values map to the range x i ∈ [0 , 1]. Furthermore, η models signalindependent read noise, h is the optical point spread function (PSF) created by the camera lens, ∗ denotes the 2-D convolution operator, and f ( ) is the camera's · response function. This image formation model assumes that the PSF is shift-invariant, but the model could be generalized to describe PSFs that vary laterally or with depth.

We assume that the camera has a linear camera response function, which is typically the case when working with raw sensor data:

$$f \left ( x _ { i } \right ) = \begin{cases} 0, & \text{if } x _ { i } < 0, \\ x _ { i }, & \text{if } 0 \leq x _ { i } \leq 1, \\ 1, & \text{if } x _ { i } > 1. \end{cases} \quad ( 2 )$$

Nonlinear camera response functions can be calibrated and inverted so as to mimic a linear response function [46]. We ignore the effects of quantization.

↦

Our goal in this work is to jointly optimize the PSF h and a reconstruction algorithm G : y → ˆ so as to x recover x from y when ‖ x ‖ ∞ glyph[greatermuch] 1. To this end, we turn to differentiable optical systems and algorithms, which we describe in the following.

## 3.1. Modeling the Optical Point Spread Function

As shown in Figure 1, our optical system is a conventional single lens reflex camera lens with a custom

Figure 2: Illustration of the proposed end-to-end optimization framework. HDR images of a training set are convolved with the PSF created by a lens surface profile φ . These simulated measurements are clipped by a function f ( ) · to emulate sensor saturation and noise η is added. The resulting RGB image y is processed by a convolutional neural network (CNN) and its output compared with the ground truth HDR image using the loss function L described in the text. In the learning stage, this loss is back-propagated into the CNN weights and bias values and also into the height values φ of the lens. During inference, a captured LDR image blurred by the optical PSF is fed directly into the pre-trained CNN to compute the reconstructed HDR image.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-46b02070-f001-4302-87e7-fcba075e716f.jpg)

diffractive optical element (DOE) add-on. Similar to a photographic filter, the DOE is mounted directly on the lens. To model the light transport from a scene, through these optical elements, to the sensor, we build on a differentiable Fourier optics model [20], an approach closely related to recent work on end-to-end camera designs [62, 8, 69, 9]. Specifically, our aim is to find the microscopic surface profile φ of the DOE that creates a PSF h which is optimally suited for the HDR image reconstruction algorithm.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-a0e772a2-1e17-49e4-a0a4-bff33880dd8e.jpg)

Assuming that the scene is at optical infinity, the complex-valued wave field of a point located on the optical axis becomes a plane wave immediately before the DOE, i.e. u in = exp( ikz ), where k = 2 π λ is the wave number and λ is the wavelength. The phase of this wave is affected by the DOE in a spatially varying manner by a complex-valued phase delay t φ , which is directly calculated from the surface profile φ as

$$\mathfrak { t } _ { \phi } ( u, v, \lambda ) = \mathbf A _ { \phi } \left ( u, v \right ) \cdot \exp \left ( i k \left ( n ( \lambda ) - 1 \right ) \phi ( u, v ) \right ). \quad \ \ F _ { \phi } \\ \mathfrak { m } _ { \phi } \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \.. \........................................................................................................................................................................................................ \.................................................................................................................................................................................................... \.. \.................................................................................................................................................................................................. \.... \.................................................................................................................................................................................................. \. \..................................................................................................................................................................................................... \. \.. \................................................................................................................................................................................................ \...... \................................................................................................................................................................................................ \... \................................................................................................................................................................................................... \... \.. \................................................................................................................................................................................................ \. \.... \................................................................................................................................................................................................ \. \. \................................................................................................................................................................................................... \. \. \.. \.............................................................................................................................................................................................. \........ \.............................................................................................................................................................................................. \..... \................................................................................................................................................................................................. \..... \.. \.............................................................................................................................................................................................. \... \.... \.............................................................................................................................................................................................. \... \. \................................................................................................................................................................................................. \... \. \.. \..........................................................................................................................................................................................$$

Here, u, v are the lateral coordinates on the DOE surface, n λ ( ) is the wavelength-dependent refractive index of the material that the DOE is made of and A φ ( u, v ) is a binary circular mask with diameter D φ that models the aperture of the DOE as

$$\mathbf A _ { \phi } ( u, v ) = \begin{cases} 1, & \text{if $u^{2}+v^{2}\leq \left(\frac{D_{\phi}}{2} \right)^{2}$}, \\ 0, & \text{otherwise.} \end{cases} \quad ( 4 ) \quad \begin{array} { c } \text{field} \\ 3. 2 \\ \end{array},$$

The wave field continues to propagate by some distance d φ to the camera lens with focal length g . This lens induces the following phase delay:

$$\mathfrak { t } _ { l } \left ( u ^ { \prime }, v ^ { \prime } \right ) = \mathbf A _ { l } \left ( u ^ { \prime }, v ^ { \prime } \right ) \cdot \exp \left ( - i \frac { k } { 2 g } \left ( u ^ { \prime 2 } + v ^ { \prime 2 } \right ) \right ). \ \ ( 5 ) \quad \text{erati}$$

Figure 3: Our training dataset consisted of HDR images scaled such that between 1 and 2% of their pixels' values were saturated and clipped. Saturated pixels are shown in red.

Although the physical compound lens contains many optical elements that correct distortions and chromatic aberrations, using a simplified thin lens model (Eq. 5) adequately describes the mathematical behavior of the lens.

Finally, the wave field propagates by a distance d s to the sensor, where its intensity h = | u sensor | 2 is recorded. Putting all of this together results in

$$\Gamma _ { \ell } \quad \text{h} _ { \phi } \left ( x, y \right ) = \left | P _ { d _ { s } } \left \{ \mathbf t _ { l } \cdot P _ { d _ { \phi } } \left \{ \mathbf t _ { \phi } \cdot e x p \left ( i k z \right ) \right \} \right \} \right | ^ { 2 }, \quad ( 6 )$$

where P d { u } models free-space propagation of a wave field u by a distance d .

## 3.2. CNN-based Image Reconstruction

To recover x from y we use a convolutional neural network based on the well-known U-Net architecture [55]. Specifically, our U-Net uses skip connections and has 5 scales with 4 consecutive downsampling operations (maxpool) and 4 consecutive upsampling operations (transposed convolutions initialized with bilinear filter weights). At each scale of the U-Net, we

Figure 4: Simulated sensor images for an example scene from our evaluation set and point spread functions (PSFs) for several different optical coding approaches: Rouf et al.'s star-shaped PSF [56] (a), our end-to-end deep optics approach applied with physically realizable constraints (b), and our end-to-end deep optics approach without physically realizable constraints applied (c). All three color channels of the PSFs are shown separately in the log domain. Whereas the star PSF continuously blurs the image along several radial streaks (a), the deep optics approach creates a PSF with several distinct peaks, which result in image content being copied at chromatically varying distances and at different intensity scales (b). The unconstrained PSF mostly blurs the green color channel while focusing the red and blue channels.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-a11f1b5b-e655-458c-9b1d-71f77f6436d2.jpg)

include one additional convolutional layer; each convolutional layer is followed by a rectified linear unit (ReLU). BatchNorm layers are used after each upsampling layer and after the final convolutional layer. This architecture is inspired by Eilertsen's work [17] but slightly leaner, which resulted in faster convergence of the lens' surface profile. As illustrated in Figure 2, each network layer has 64 feature maps.

## 3.3. End-to-end Training Details

We jointly optimize the PSF and the CNN via the end-to-end (E2E) framework illustrated in Figure 2. In particular, using thousands of HDR training images, we simulate (1) passing an HDR image through our optical system, (2) capturing a noisy and saturated LDR image with a sensor, and (3) reconstructing the HDR images with the CNN. We compute the corresponding loss and use Tensorflow's autodifferentiation capabilities to back-propagate the error and update the parameters θ of the CNN and the height map φ of the PSF.

Loss Function Following Kalantari and Ramamoorthi [31], we originally experimented with minimizing the glyph[lscript] 2 -loss between the tone-mapped reconstruction and ground-truth HDR images. While successful in training the CNN, this loss function caused the network to focus its efforts on the most overexposed and challenging images in the training data. Using this approach, a typical Dirac delta-type was found as a locally optimal solution for the PSF.

To encourage the development of more powerful PSFs, we instead minimize the sum, over all the batches, of per-batch, γ -corrected, glyph[lscript] 2 -loss:

$$\mathcal { L } _ { \text{Data} } = \sum _ { \text{B} \subset \text{Batches} } \left \| ( x _ { B } + \epsilon ) ^ { \gamma } - ( \hat { \mathbf x } _ { B } + \epsilon ) ^ { \gamma } \right \| _ { 2 },$$

where γ = 1 2 and / glyph[epsilon1] is a small constant we add to avoid the non-differentiability around 0. That is, we minimized the root mean-squared-error (RMSE) over batches, as opposed to the typical mean-squared-error (MSE) loss.

In the context of regression, sums of glyph[lscript] 2 -norms over groups encourage group-sparse solutions [61]. In our context, sums of glyph[lscript] 2 -norms make the network more robust to outliers, i.e. it allows the network to fail for the most challenging reconstructions so long as the glyph[lscript] 2 -norm of the error is small for most batches.

Incorporating Fabrication Constraints To ensure that the optimized height map can be manufactured, we clip its values to the maximum range during training and add an additional smoothness term on φ to prevent the resulting surface profile to include many discontinuities. Specifically, we add the loss

$$\mathcal { L } _ { \text{Reg} } = \nu \| \mathbf D * \phi \| _ { 2 } ^ { 2 },$$

to the overall loss function, where D is a Laplacian filter and ν = 10 9 is a weighting parameter.

Datasets Following Eilertsen et al. [17], we use training and validation datasets consisting of 2837

Table 1: Quantitative evaluation for the entire test set. Several single-shot HDR imaging approaches are compared using a perceptual image difference computed by HDR-VDP-2 and peak signal-to-noise ratio (PSNR) computed in the linear domain (L) and in the γ -corrected domain.

|                                 |   HDR-VDP-2 |   PSNR-L |   PSNR- γ |
|---------------------------------|-------------|----------|-----------|
| LDR                             |        51.4 |     39.8 |      38.9 |
| HDR-CNN [17]                    |        58.6 |     42.1 |      42.9 |
| U-Net                           |        56.4 |     41.8 |      42.3 |
| Star PSF + U-Net[56]            |        56.7 |     42.1 |      42.3 |
| E2E PSF + U-Net                 |        60.6 |     45.6 |      44.3 |
| E2E PSF + U-Net (unconstrained) |        67.1 |     46.8 |      40.7 |

HDR images drawn from a combination of videos and images from a number of sources. We performed data augmentation by cropping, rescaling, and adjusting hue and saturation. The final training set contains just under 60,000 different HDR images with a resolution of 320 × 320 pixels. Our test set consists of 223 HDR images, of which 83 are still images and the rest frames drawn from every 10th frame of four separate video sequences (which were not used for training). To generate LDR/HDR image pairs, we simulated capturing LDR frames where we set the exposure such that between 1 and 2% of the pixels were saturated. A histogram of the pixel values in our training data is shown in Figure 3.

Miscellaneous Training Details We trained the end-to-end model using the Adam optimizer with a minibatch size of 8. We applied an exponential learning rate decay with an initial rate of 0 0001. . We trained the network for 100 epochs, which took about 3 days on a Pascal Titan X graphics processing unit.

Source code, trained models, and datasets will be made available.

## 4. Analysis and Evaluation

In this section, we evaluate the proposed method in simulation and show comparisons to various other single-shot HDR imaging approaches.

Figure 4 shows simulated sensor images and PSFs for several options of optically coding the sensor image before reconstruction. The PSF that was optimized with the method proposed in the previous section is shown in (b) and also in Figure 6. This PSF contains several peaks, each creating a shifted and scaled copy of the sensor image superimposed on itself. In this way, the PSF serves to multiplex together different exposures of the image. The copies for individual color channels appear at slightly different location, which leads to visible chromatic aberrations in the sensor image; such chromatic aberrations are inevitable when using a single DOE design. We also optimize a PSF using the proposed optimization method but without parameterizing it by a physically realizable optical element (c). Therefore, the color channels of this PSF are completely independent of one another and individual pixel values can be arbitrary as long as the are in the range [0 , 1] and sum to 1 for each channel. We also show the star-shaped PSF proposed by Rouf et al. [56].

We compare several reconstruction approaches in Figure 5. These include the conventional LDR image, Eilertsen et al.'s CNN applied to this LDR image (HDR-CNN), the proposed smaller U-Net applied to this LDR image, the U-Net applied to an image captured with the star PSF, and our end-to-end deep optics approach with physically realizable constraints. For fair comparison, the U-Nets in each example are trained for the respective PSF. We show regions of interest in the insets along with visible differences predicted by HDR-VDP-2 [40]. In all cases, the proposed deep optics approach achieves the best results.

Finally, we show a quantitative comparison of all these methods in Table 1. Here, we report the average perceptual difference as computed by HDR-VDP-2 as well as peak signal-to-noise ratio (PSNR) over the entire test set in the linear and γ -corrected domains. We observed that the end-to-end (E2E) approaches achieve the best image quality. The unconstrained E2E approach is usually better than the physically realizable version because it has more degrees of freedom. However, the PSNRγ of the unconstrained PSF is slightly lower than that of the realizable approach. This is likely due to the fact that the network was trained using an outlier-robust loss, to which it over-fit. Recall, optimizing PSNR directly lead to sub-optimal convergence of the PSF (see Sec. 3.3).

## 5. Fabrication and Implementation

Lens Fabrication Once a phase profile is optimized, we fabricate the corresponding diffractive optical element (DOE) using polydimethyl-siloxane (PDMS) through replica molding. Figure 6 shows the optimized height profile (left) along with a 3D rendering of profilometer measurements of the fabricated DOE (center), which has a diameter of 5 mm. Qualitatively, the

Figure 5: Comparison of various single-shot HDR imaging approaches. In all examples, the whole images are displayed at -1 stop and regions of interest at -3 stops. In the columns, we show the ground truth HDR image, a corresponding LDR image, Eilertsen et al.'s CNN-based reconstruction applied to the LDR image (HDR-CNN) [17], a slightly simpler U-Net applied to the LDR image, the star-shaped PSF proposed by Rouf et al. [56] with a U-Net reconstruction, and our end-to-end deep optics approach (E2E). In all three examples, the peak signal-tonoise ratio (PSNR) and also observed image quality (see insets) are best for our approach. Color-coded insets show probabilities of perceiving the difference between the reconstructions and the ground truth HDR images, as computed by the HDR-VDP-2 visible differences predictor. Again, our approach qualitatively and quantitatively (evaluated with HDR-VDP-2 Q value) outperforms other approaches.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-cda837c4-ffbe-4cde-9600-a0f099dc5be5.jpg)

shapes are similar. We also show the simulated (top right) and the captured PSFs (bottom right). These PSFs match well and both create a Dirac peak in the center and lower-amplitude satellite peaks at slightly different locations for the three color channels. This PSF is created by the lens surface profile that resembles a grating-like structure. The captured PSF is slightly blurrier than the simulation and there is additional glare, both likely due to slight fabrication errors and interreflections between the lens elements.

System Integration We mount the fabricated DOE as an add-on to a conventional single lens reflex camera (Canon Rebel T5) equipped with a standard compound lens (Nikon Nikkor 35 mm). The DOE is fixed in a Thorlabs lens tube with rotating optic adjustment (SM1M05), which is coupled to the SLR lens via an optomechanical adapter (Thorlabs SM1A2). Figure 1

Figure 6: Optimized height profile of the diffractive optical element (DOE, left) along with profilometer measurements of the fabricated DOE. The DOE structure partially resembles that of a grating, which creates multiple peaks in the point spread function (PSF, right). Intuitively, this PSF creates three shifted and scaled copies of the input image. Although the measured PSF is slightly blurrier than the simulated PSF, likely due to imperfections in the fabrication process and approximations of our image formation model, their general shapes are comparable.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-e7923cc5-1960-457a-82d5-4c503e0e9460.jpg)

(right) shows the DOE and SLR lens. In this setup, the DOE is physically mounted at a slight distance to the compound camera lens. The exact distance between these optical elements is unknown because we model the primary camera lens as having a single refractive surface. While this is the easiest approach, a more detailed optical model of the compound lens may be desirable, although to model it appropriately, proprietary information from the lens manufacturer would have to be known. Moreover, the lack of anti-reflection coatings on the DOE may add interreflections and glare, and likely contributes to a slight mismatch between simulated and captured PSFs.

To calibrate our camera system, we capture several different exposures of a white light source behind a 75 µ m-sized pinhole. These photographs are merged into a single HDR image using the technique described by Debevec [15]. This captured point spread function of our optical system is used to refine the pre-trained CNN by optimizing its parameters for the fixed captured PSF, as described in Section 3.3. Refining the CNN with a fixed PSF is significantly faster than training the end-to-end system from scratch and only takes a few hours.

## 6. Experimental Results

Using the prototype camera described in the previous section, we captured several HDR example scenes (see Figs. 1, 7). These include three scenes recorded in a laboratory setting (Fig. 1 and Fig. 7, top and center row) and one outdoor scene captured at night (Fig. 7, bottom row). In Figure 7, captured measurements along with reconstructions computed by our CNN as well as reference LDR and HDR images and the result of the HDR-CNN [17] are shown. In all of these examples, the captured LDR images include saturated areas that actually contain detail, which is completely lost in the measurements, such as the filament of a light bulb (top row) or the structure of a light source on a wall (bottom row). We show these images as well as magnified closeups (right column) at varying exposure values (EVs) to best highlight these details. In should be noted that these are all examples where we expect the HDR-CNN approach to fail, because the network simply has no information about the detail in the saturated parts-the best it can do is to inpaint these parts. The inpainting process results in smooth regions that exceed the dynamic range of the LDR sensor image but that do not resemble the actual content in these examples.

## 7. Discussion

In summary, we propose an end-to-end approach to jointly training an optical encoder, i.e. the point spread function created by a custom optical element, and electronic decoder, i.e. a convolutional neural network, for the application of single-shot high-dynamicrange imaging. As opposed to CNN-based methods that operate directly on conventional LDR images, our deep optics approach has the ability to optically encode details from bright parts of the scene into the LDR measurements. In particular, our method uses a unique multiplexing solution to HDR imaging, which it developed automatically, wherein the PSF superimposes multiple shifted exposures on top of one another. The proposed framework builds on the emerging idea of end-to-end optimization of optics and image processing, but to our knowledge it is the first to explore this general methodology for single-shot HDR imaging.

Limitations Conceptually, HDR-CNN approaches that operate directly on conventional LDR images solve an inpainting problem, which is ill-posed and often fails. Accordingly, and unlike our deep optics approach, the worst case solution of HDR-CNNs is a conventional LDR image, which is directly recorded and which may not always need additional post-processing to begin with. Our method changes the optical image formation, so post-processing becomes a necessary part of the imaging pipeline. Slight reconstruction artifacts in our experimental results (Figs. 1, 7), which are primarily due to imperfections in the PSF calibration, can be found in both non-saturated and saturated parts of the

LDR

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-c3f86ed5-0b3a-4f51-a52b-0d8adcb3c6ad.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-885294a1-92e9-4e69-bef4-f918aa69a698.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-bee2dda1-c766-4a42-8187-af681687e32f.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-43b273ef-ba7e-46c9-b3eb-06b817f9b1e8.jpg)

E2E Reconstruction Reference HDR

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-622b46d8-3523-4540-ac61-7651eaa976ff.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-ee61501c-85f7-4705-a179-f455cc2af895.jpg)

-2 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-46044406-e533-47aa-9ea1-8bf6c91ec7d1.jpg)

-2.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-ec1b32cd-33cc-4e1f-8f39-6085f204de0b.jpg)

-1.3 EV

HDR-CNN

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-670fc933-ff37-4b73-96da-c3d2f87419e3.jpg)

-2 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-a34646bb-f245-4815-a1f3-effbfeb70da4.jpg)

-2.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-dea68835-c5a8-443a-9661-124386869a4d.jpg)

-1.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-b41929e4-6f41-4ca5-ae64-cf576a551d92.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-877dc41b-9fd5-4c32-828a-62d07a6afb33.jpg)

0 EV

-1.6 EV

0 EV

-1.6 EV

-1.6 EV

-4.9 EV

-2 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-29709b82-4c59-41b4-8b53-c0d802308dcc.jpg)

-2.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-5a775ac1-7ceb-4a52-b0f3-ed8edd3cd381.jpg)

-1.3 EV

E2E Measurement

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-0179a1d2-23a6-4cb3-b26d-a433abf90000.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-66bf4db0-251f-4746-916a-b06c7776cb3c.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-ce2e652f-5970-42b3-97b9-e5f251534d15.jpg)

0 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-048c2305-95d2-4a68-86fd-67384549d211.jpg)

-2.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-d4927437-1d67-4c06-a200-09b7b5461901.jpg)

-3.3 EV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-b3cd228f-dd6a-4729-bb77-603bfdc1c8f3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-3202d05e-956c-47c1-a844-2eac6cf148fb.jpg)

-4.3 EV

Figure 7: Experimental results of three indoor scenes (top three rows) and one outdoor scene at night (bottom row). The limited dynamic range of the sensor loses details in the brighter parts of the captured LDR image (first column) as compared with the reference HDR image (fifth column). A CNN operating directly on the LDR images hallucinates brighter content in these saturated parts, but it is missing the detail (second column). The measurements captured with our prototype camera optically encode this detail in the image by superimposing several scaled and shifted copies of the image on itself (third column). This information is used by our CNN to recover the missing parts of the scene while digitally removing the image copies (fourth column). The closeups, showing E2E measurements, HDR-CNN, E2E Reconstruction, and ground truth HDR, demonstrate that deep optics is more successful in recovering bright detail of HDR scenes than other single-shot HDR imaging approaches (right column).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-704892a4-0e31-4489-935e-487e30ad3345.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00620v1-figure-fc7110c4-6b9b-44e7-8603-8ce03a2f9d97.jpg)

image due to the need for processing the entire LDR image, rather than just its saturated parts.

In essence, the inverse problem in our method is more closely related to deconvolution problems than to inpainting problems, although for extremely bright scenes where even the lower intensity copies of the sensor image saturate, inpainting is unavoidable and the additional scene copies in our measurements may ac- tually be harmful. Therefore, careful curation of the training set is crucial, because it needs to include HDR images with values adequately representing those observed during inference. A network is likely going to fail to produce high-quality results for conditions that it has not been trained for.

The fabricated diffractive optical element creates a PSF that closely resembles the simulated PSF. Yet,

blur and glare, likely due to interreflections between optical elements, are problematic. Optical blur makes the deconvolution problem harder and glare causes the PSF to be slightly shift variant, which limits the effective field of view of our captured data (depth-of-field is unaffected). Thus, although the add-on approach of mounting the DOE in front of an SLR camera lens is convenient and flexible, integrating the DOE into the aperture plane of the primary lens may produce better results, as it more closely resembles our paraxial image formation model. Alternatively, the image formation model could be generalized to a non-paraxial model. Finally, anti-reflection coatings may help mitigate glare in the optics.

Future Work The end-to-end methods enable designing tailored optics for a particular task, rather than just capturing the sharpest image. Evaluating the benefits of end-to-end optimization of optics and image processing for other applications, including multispectral imaging, light field imaging, lensless imaging, and computational microscopy, is an interesting avenue of future work.

## 8. Conclusion

End-to-end design of optics and image reconstruction algorithms is an emerging paradigm for designing domain-specific computational cameras. We demonstrate clear benefits of this approach to the longstanding problem of single-shot HDR imaging. Beyond this application, deep computational cameras have the potential to re-define the next generation of domainspecific cameras that capture richer visual information than conventional cameras and that enable entirely new imaging modalities in a wide range of applications.

## References

- [1] Manoj Aggarwal and Narendra Ahuja. Split aperture imaging for high dynamic range. International Journal of Computer Vision , 58(1):7-17, 2004.
- [2] N. Antipa, S. Necula, R. Ng, and L. Waller. Single-shot diffuser-encoded light field imaging. In Proc. IEEE ICCP , pages 1-11, 2016.
- [3] M. S. Asif, A. Ayremlou, A. Sankaranarayanan, A. Veeraraghavan, and R. G. Baraniuk. Flatcam: Thin, lensless cameras using coded aperture and computation. IEEE Trans. Computational Imaging , 3(3):384397, 2017.
- [4] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers. Advanced High Dynamic Range Imaging: Theory and Practice . AK Peters (CRC Press), 2011.
- [5] Francesco Banterle, Patrick Ledda, Kurt Debattista, and Alan Chalmers. Inverse tone mapping. In Proc. GRAPHITE , pages 349-356, 2006.
- [6] D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller. Multiscale gigapixel photography. Nature , 486:386389, 2012.
- [7] Ayan Chakrabarti. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems , pages 3081-3089, 2016.
- [8] Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. Hybrid opticalelectronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports , 8(1):12324, 2018.
- [9] Julie Chang and Gordon Wetzstein. Deep optics for monocular depth estimation and 3d object detection. In Proc. ICCV , 2019.
- [10] Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. High-quality hyperspectral reconstruction using a spectral prior. ACM Trans. Graph. (SIGGRAPH Asia) , 36(6):218:1-218:13, 2017.
- [11] O. Cossairt, D. Miau, and S.K. Nayar. Gigapixel computational imaging. In Proc. ICCP , 2011.
- [12] Oliver Cossairt and Shree Nayar. Spectral focal sweep: Extended depth of field from chromatic aberrations. In Proc. ICCP , pages 1-8, 2010.
- [13] Oliver Cossairt, Changyin Zhou, and Shree Nayar. Diffusion coded photography for extended depth of field. ACM Trans. Graph. (SIGGRAPH) , 29(4):31:1-31:10, 2010.
- [14] P. Debevec. Image-based lighting. IEEE Computer Graphics and Applications , 22(2):26-34, 2002.
- [15] Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In Proc. SIGGRAPH , pages 369-378, 1997.
- [16] Edward R. Dowski and W. Thomas Cathey. Extended depth of field through wave-front coding. OSA Appl. Opt. , 34(11):1859-1866, 1995.
- [17] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafaglyph[suppress] l K Mantiuk, and Jonas Unger. Hdr image reconstruction from a single exposure using deep cnns. ACM Transactions on Graphics (TOG) , 36(6):178, 2017.
- [18] Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. Deep reverse tone mapping. ACM Trans. Graph. , 36(6):177:1-177:10, 2017.
- [19] Orazio Gallo, Natasha Gelfandz, Wei-Chao Chen, Marius Tico, and Kari Pulli. Artifact-free high dynamic range imaging. In Proc. ICCP , pages 1-7, 2009.
- [20] Joseph W Goodman. Introduction to Fourier optics . Roberts and Company Publishers, 2005.
- [21] Miguel Granados, Kwang In Kim, James Tompkin, and Christian Theobalt. Automatic noise modeling for ghost-free hdr reconstruction. ACM Trans. Graph. , 32(6):201:1-201:10, 2013.

| [22]   | H. Haim, S. Elmalem, R. Giryes, A. M. Bronstein, and E. Marom. Depth estimation from a single image using deep learned phase coded mask. IEEE Trans. Computational Imaging , 4(3):298-310, 2018.                                                                          |
|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [23]   | Saghi Hajisharif, Joel Kronander, and Jonas Unger. Adaptive dualiso hdr reconstruction. EURASIP Jour- nal on Image and Video Processing , 2015, 2015.                                                                                                                     |
| [24]   | S. W. Hasinoff, F. Durand, and W. T. Freeman. Noise- optimal capture for high dynamic range photography. In Proc. CVPR , pages 553-560, 2010.                                                                                                                             |
| [25]   | Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, An- drew Adams, Jonathan T Barron, Florian Kainz, Ji- awen Chen, and Marc Levoy. Burst photography for high dynamic range and low-light imaging on mo- bile cameras. ACM Transactions on Graphics (TOG) , 35(6):192, 2016. |
| [26]   | Lei He, Guanghui Wang, and Zhanyi Hu. Learning depth from single images with deep neural network embedding focal length. IEEE Transactions on Image Processing , 27(9):4676-4689, 2018.                                                                                   |
| [27]   | Eran Hershko, Lucien E. Weiss, Tomer Michaeli, and Yoav Shechtman. Multicolor localization microscopy and point-spread-function engineering by deep learn- ing. OSA Opt. Express , 27(5):6158-6183, 2019.                                                                 |
| [28]   | Roarke Horstmeyer, Richard Y Chen, Barbara Kappes, and Benjamin Judkewitz. Convolutional neu- ral networks that teach microscopes how to image. arXiv preprint arXiv:1709.07223 , 2017.                                                                                   |
| [29]   | J. Hu, O. Gallo, K. Pulli, and X. Sun. Hdr deghosting: How to deal with saturation? In Proc. CVPR , 2013.                                                                                                                                                                 |
| [30]   | Daniel S Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H Kim. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Transactions on Graphics (TOG) , 38(4):117, 2019.                                               |
| [31]   | Nima Khademi Kalantari and Ravi Ramamoorthi. Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. (SIGGRAPH) , 36(4), 2017.                                                                                                                               |
| [32]   | Nima Khademi Kalantari, Eli Shechtman, Connelly Barnes, Soheil Darabi, Dan B Goldman, and Pradeep Sen. Patch-based High Dynamic Range Video. ACM Trans. Graph. (SIGGRAPH Asia) , 32(6), 2013.                                                                             |
| [33]   | Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. High dynamic range video. ACM Trans. Graph. (SIGGRAPH) , 22(3):319- 325, 2003.                                                                                                                   |
| [34]   | Michael Kellman, Emrah Bostan, Michael Chen, and Laura Waller. Data-driven design for fourier ptycho- graphic microscopy. In 2019 IEEE International Con- ference on Computational Photography (ICCP) , pages 1-8. IEEE, 2019.                                            |
| [35]   | E. A. Khan, A. O. Akyuz, and E. Reinhard. Ghost removal in high dynamic range images. In Proc. ICIP , pages 2005-2008, 2006.                                                                                                                                              |
| [36]   | Siyeong Lee, Gwon Hwan An, and Suk-Ju Kang. Deep recursive hdri: Inverse tone mapping using generative adversarial networks. In Proc. ECCV , pages 596-611, 2018.                                                                                                         |

- [37] Anat Levin, Rob Fergus, Fr´ edo Durand, and William T. Freeman. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. (SIGGRAPH) , 26(3), 2007.
- [38] Anat Levin, Samuel W. Hasinoff, Paul Green, Fr´ edo Durand, and William T. Freeman. 4d frequency analysis of computational cameras for depth of field extension. ACM Trans. Graph. (SIGGRAPH) , 28(3):97:197:14, 2009.
- [39] Mann, Picard, S. Mann, and R. W. Picard. On being 'undigital' with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of IS&amp;T , pages 442-448, 1995.
- [40] Rafat Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. (SIGGRAPH) , 30(4):40:1-40:14, 2011.
- [41] Julio Marco, Quercus Hernandez, Adolfo Mu˜oz, Yue n Dong, Adrian Jarabo, Min H. Kim, Xin Tong, and Diego Gutierrez. Deeptof: Off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) , 36(6):219:1-219:12, 2017.
- [42] Kshitij Marwah, Gordon Wetzstein, Yosuke Bando, and Ramesh Raskar. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Trans. Graph. (SIGGRAPH) , 32(4):46:1-46:12, 2013.
- [43] M. McGuire, W. Matusik, H. Pfister, B. Chen, J. F. Hughes, and S. K. Nayar. Optical splitting trees for high-precision monocular imaging. IEEE Computer Graphics and Applications , 27(2):32-42, 2007.
- [44] Tom Mertens, Jan Kautz, and Frank Van Reeth. Exposure fusion: A simple and practical alternative to high dynamic range photography. In Computer graphics forum (Eurographics) , volume 28, pages 161-171. Wiley Online Library, 2009.
- [45] Laurence Meylan, Scott Daly, and Sabine S¨sstrunk. u The reproduction of specular highlights on high dynamic range displays. IS&amp;T/SID 14th Color Imaging Conference (CIC) , 2006.
- [46] T. Mitsunaga and S. K. Nayar. Radiometric self calibration. In Proc. IEEE CVPR , volume 1, pages 374380, 1999.
- [47] S. K. Nayar and T. Mitsunaga. High dynamic range imaging: spatially varying pixel exposures. In Proc. CVPR , volume 1, pages 472-479 vol.1, 2000.
- [48] Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Tomer Michaeli, and Yoav Shechtman. Dense three dimensional localization microscopy by deep learning. arXiv preprint arXiv:1906.09957 , 2019.
- [49] Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz, and Pat Hanrahan. Light field photography with a hand-held plenoptic camera. Tech Report CSTR 2005-02, 2005.
- [50] Sri Rama Prasanna Pavani, Michael A. Thompson, Julie S. Biteen, Samuel J. Lord, Na Liu, Robert J.

- Twieg, Rafael Piestun, and W. E. Moerner. Threedimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. 106(9):2995-2999, 2009.
- [51] Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Trans. Graph. (SIGGRAPH) , 35(4):31:1-31:11, 2016.
- [52] Ramesh Raskar, Amit Agrawal, and Jack Tumblin. Coded exposure photography: Motion deblurring using fluttered shutter. ACM Trans. Graph. (SIGGRAPH) , 25(3):795-804, 2006.
- [53] Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. High dynamic range imaging: acquisition, display, and image-based lighting . Morgan Kaufmann, 2010.
- [54] Allan G. Rempel, Matthew Trentacoste, Helge Seetzen, H. David Young, Wolfgang Heidrich, Lorne Whitehead, and Greg Ward. Ldr2hdr: On-the-fly reverse tone mapping of legacy video and photographs. ACM Trans. Graph. (SIGGRAPH) , 26(3), 2007.
- [55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.
- [56] Mushfiqur Rouf, Rafaglyph[suppress] l Mantiuk, Wolfgang Heidrich, Matthew Trentacoste, and Cheryl Lau. Glare encoding of high dynamic range images. In Proc. CVPR 2011 , pages 289-296, 2011.
- [57] Helge Seetzen, Wolfgang Heidrich, Wolfgang Stuerzlinger, Greg Ward, Lorne Whitehead, Matthew Trentacoste, Abhijeet Ghosh, and Andrejs Vorozcovs. High dynamic range display systems. ACM Trans. Graph. (SIGGRAPH) , 23(3):760-768, 2004.
- [58] Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. Robust patch-based hdr reconstruction of dynamic scenes. ACM Trans. Graph. (SIGGRAPH Asia) , 31(6):203:1-203:11, 2012.
- [59] Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. Convolutional sparse coding for high dynamic range imaging. In Computer Graphics Forum (Eurographics) , pages 153-163, 2016.
- [60] Yoav Shechtman, Steffen J. Sahl, Adam S. Backer, and W. E. Moerner. Optimal point spread function design for 3d imaging. Phys. Rev. Lett. , 113:133902, 2014.
- [61] Noah Simon, Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A sparse-group lasso. Journal of Computational and Graphical Statistics , 22(2):231245, 2013.
- [62] Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging.
- ACM Transactions on Graphics (TOG) , 37(4):114, 2018.
- [63] Shuochen Su, Felix Heide, Gordon Wetzstein, and Wolfgang Heidrich. Deep end-to-end time-of-flight imaging. In Proc. CVPR , 2018.
- [64] Michael D. Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. A versatile hdr video production system. ACM Trans. Graph. (SIGGRAPH) , 30(4):41:1-41:10, 2011.
- [65] Alex Turpin, Ivan Vishniakou, and Johannes D Seelig. Light scattering control with neural networks in transmission and reflection. Arxiv: 180505602 [Cs] , 2018.
- [66] Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan, and Jack Tumblin. Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. (SIGGRAPH) , 26(3), 2007.
- [67] Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. Single disperser design for coded aperture snapshot spectral imaging. OSA Appl. Opt. , 47(10):B44-B51, 2008.
- [68] G. Wetzstein, I. Ihrke, and W. Heidrich. Sensor saturation in fourier multiplexed imaging. In Proc. CVPR , pages 545-552, 2010.
- [69] Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. Phasecam3d - learning phase masks for passive single view depth estimation. In Proc. ICCP , 2019.
- [70] H. Zhao, B. Shi, C. Fernandez-Cull, S. Yeung, and R. Raskar. Unbounded high dynamic range photography using a modulo camera. In Proc. ICCP , pages 1-10, 2015.
- [71] C. Zhou, S. Lin, and S. K. Nayar. Coded aperture pairs for depth from defocus. In Proc. ICCV , 2009.
- [72] C. Zhou and S. Nayar. What are good apertures for defocus deblurring? In Proc. IEEE ICCP , pages 1-8, 2009.
Christopher A. Metzler Hayato Ikoma Yifan Peng Gordon Wetzstein
Deep Optics for Single-shot High-dynamic-range Imaging
High-dynamic-range (HDR) imaging is crucial for many computer graphics and
vision applications. Yet, acquiring HDR images with a single shot remains a
challenging problem. Whereas modern deep learning approaches are successful at
hallucinating plausible HDR content from a single low-dynamic-range (LDR)
image, saturated scene details often cannot be faithfully recovered. Inspired
by recent deep optical imaging approaches, we interpret this problem as jointly
training an optical encoder and electronic decoder where the encoder is
parameterized by the point spread function (PSF) of the lens, the bottleneck is
the sensor with a limited dynamic range, and the decoder is a convolutional
neural network (CNN). The lens surface is then jointly optimized with the CNN
in a training phase; we fabricate this optimized optical element and attach it
as a hardware add-on to a conventional camera during inference. In extensive
simulations and with a physical prototype, we demonstrate that this end-to-end
deep optical imaging approach to single-shot HDR imaging outperforms both
purely CNN-based approaches and other PSF engineering approaches.