## Single-Shot High Dynamic Range Imaging with Spatially Varying Exposures Considering Hue Distortion

Chihiro Go, Yuma Kinoshita, Sayaka Shiota and Hitoshi Kiya

Tokyo Metropolitan University , Tokyo, Japan

E-mail: go-chihiro@ed.tmu.ac.jp, kinoshita-yuma@ed.tmu.ac.jp, sayaka@tmu.ac.jp, kiya@tmu.ac.jp

Abstract -We proposes a novel single-shot high dynamic range imaging scheme with spatially varying exposures (SVE) considering hue distortion. Single-shot imaging with SVE enables us to capture multi-exposure images from a single-shot image, so high dynamic range images can be produced without ghost artifacts. However, SVE images have some pixels at which a range supported by camera sensors is exceeded. Therefore, generated images have some color distortion, so that conventional imaging with SVE has never considered the influence of this range limitation. To overcome this issue, we consider estimating the correct hue of a scene from raw images, and propose a method with the estimated hue information for correcting the hue of SVE images on the constant hue plain in the RGB color space.

Index Terms -high dynamic range imaging, spatially varying exposures, maximally saturated color

## I. INTRODUCTION

Fig. 1. SVE image

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00186v1-figure-773a3292-bc2b-4593-a6dc-b3a164d1f0bf.jpg)

A conceptual diagram of the RGB color space Fig. 2. Constatnt hue plane with RGB color space

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00186v1-figure-8fac7b68-29dd-4c86-8e29-bf09b2fac4ac.jpg)

I

The low dynamic range (LDR) imaging sensors used in modern digital cameras cannot express the dynamic range of a real scene, due to a limited dynamic range which imaging sensors have [1]-[3]. The limitation results in the low contrast of images taken by digital cameras. The most common approach for HDR imaging is to fuse multi-exposure images which are to merge a set of LDR images taken with different exposure times. This approach requires to capture multi-exposure images by taking at the different time, so there are ghost artifact issues, due to the movement of the camera and the subject. One of ghost-free techniques for HDR imaging is to employ spatially varying exposures (SVE) [4]-[7]. In the SVEbased imaging, a scene is captured with varying exposures for each pixel in a single image, and multiple sub-images with each exposure are obtained. However, conventional SVE-based methods focus on the luminance of a scene, so they cause color distortion, due to the influence of the limited dynamic range. Fig. 1 . (d) The 8-bit color RGB values rived from 𝒌

C i ( p ) of the LDR image C i ( p ) = round( C f · 255) ) rounds A to its nearest integer value, and two images according to exposure values. An interpolation operation is applied to each raw image for producing two raw images with the same size as X X : low and X high . However, X has some pixels at which a range supported by the camera sensor is exceeded.

is d

(

C

i

(

p

)

(

L

To overcome this issue, a novel single-shot imaging scheme with SVE is proposed in this paper. The correct hue of a scene is estimated from raw images, and then the estimated hue information is employed on the constant hue plain in the RGB color space [8] for correcting the hue of SVE images. where round( A { R p ,G i ( ) i ( p , B ) follows:

## i ( p ) } . Further, the value of C i B. Constant hue place in the RGB color space

p

)

is redefined

## II. RELATED WORKS

## A. SVE image

C

i

(

A raw Bayer image X sensed with SVE sensor is illustrated in Fig.1, where the exposure value alternates every two lines in the Bayer image. The raw image X is separated into where if C i ( p )

(

Als if

C

p ) =    0 ( C i ( p ) &lt; 0) C i ( p ) (0 ≤ C i ( p ) ≤ 255) 255 (255 &lt; C i ( p )) takes a value over 255, it is clipped at 255. An input image is a 24-bit full color image and each pixel of the image is represented as x ∈ [0 , 1] 3 . x , x r g and x b are the R, G, and B components of the pixel x , respectively, as shown in Fig.2. In the RGB color space, a set of pixels which has the same hue forms a plane, called constant hue plane [8]. The shape of the constant hue plane is the triangle whose vertices correspond to white, black and the maximally saturated color, where w = (1 1 1) , , , k = (0 0 0) , , and c are white, black

i

(

p

)

takes a negative value, it

is clipped at 0.

The roundi quantization in Eq.(3) and clipping in Eq.(4) generate some error.

2.2.

Constant hue plane in the RGB color space

We focus on the constant hue plane in the RGB color space [12]

discuss color distortion. An input image is a 24-bit full color ima

&lt;

l

a

t

e

x

i

s

h

1

\_

b

6

4

=

"

/

g

Z

z

D

f

N

o

q

d

U

n

m

p

P

9

E

R

W

c

0

Q

&gt;

A

C

j

V

S

8

F

2

X

7

K

r

J

Y

H

B

k

I

O

5

3

+

L

y

u

v

w

G

T

M

&lt;

l

a

t

e

x

i

s

h

1

\_

b

6

4

=

"

Y

+

3

z

G

d

U

u

c

m

r

T

M

w

8

q

N

W

f

&gt;

A

C

j

n

V

E

9

S

Q

F

D

2

X

/

7

K

o

J

H

P

R

B

L

O

g

0

v

p

k

5

Z

y

I

&lt;

l

a

t

e

x

i

s

h

1

\_

b

6

4

=

"

J

j

3

n

p

L

m

d

7

2

c

O

W

+

8

R

M

&gt;

A

C

H

V

S

N

Q

E

G

P

g

U

B

Z

q

I

X

k

f

9

F

T

D

0

u

w

z

r

5

y

o

Y

/

v

K

&lt;

l

a

t

e

x

i

s

h

1

\_

b

6

4

=

"

V

U

c

p

g

D

q

M

5

N

R

z

o

3

u

I

&gt;

A

C

H

F

L

w

G

9

k

d

0

m

j

T

/

Q

P

+

Y

Z

W

f

J

n

7

S

2

r

X

O

B

v

K

E

y

8

&lt;

l

a

t

e

x

i

s

h

1

\_

b

6

4

=

"

A

F

D

M

p

f

z

m

q

G

g

v

U

0

T

L

c

V

&gt;

C

o

X

H

S

N

B

3

8

d

J

y

Z

W

9

Q

O

u

K

+

j

/

R

n

w

I

r

Y

5

7

P

2

E

k

Fig. 3. Outline of proposed method

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00186v1-figure-00c9d9e3-2f7f-4315-8a0c-6d2273d88bd7.jpg)

and the maximally saturated color with the same hue as x , respectively. The maximally saturated color c = ( c r , c g , c b ) is calculated by, under l = { r, g, b }

$$c _ { l } = \frac { x _ { l } - \min ( x ) } { \max ( x ) - \min ( x ) } \quad \quad \ \ ( 1 ) \ \text{and} \ \text{the p}$$

where max( ) · and min( ) · are functions that return the maximum and minimum elements of the pixel x , respectively.

A pixel x can also be represented as a linear combination as

$$x = a _ { w } w + a _ { k } k + a _ { c } c \, \quad \ \ ( 2 ) \ \ \ was \,.$$

where the coefficient meet the equations,

$$a _ { w } + a _ { k } + a _ { c } = 1, \quad \quad \ \ ( 3 ) \quad _ { \Delta \ T }$$

$$0 \leq a _ { w }, a _ { k }, a _ { c } \leq 1. \quad \quad \ ( 4 ) \quad \ \varepsilon \varepsilon$$

This method is applied to various methods [9], [10].

## III. PROPOSED METHOD

The outline of the proposed method is shown in Fig.3.

## A. Procedure

- 1) Separation and interpolation: A raw image X is first divided into two raw images, according to the exposure value. Next, interpolation processing is applied to each raw image for producing two raw images: X low and X high
- 2) Exposure compensation: Perform scene-segmentation based exposure compensation [3].
- 3) Demosaicing: A demosaicing algorithm is applied to the compensated images to obtain RGB images.
- 4) Image fusion: The RGB images are fused by using a fuse function.
- 5) Hue compensation: The hue of the fused image Y out is compensated by using the proposed method as shown below.
- B. Hue estimation and hue correction

From Eq.(2), a pixel value y out in Y out is given by,

$$y _ { o u t } = a _ { w } w + a _ { k } k + a _ { c } c _ { o u t }, \text{ \quad \ \ } ( 5 ) \text{ \ large}$$

where c out is the maximally saturated color of y out , and a w , a k , a c and c out are calculated from y out . Note that c out may be distorted due to the influence of some pixels at which a range supported by the camera sensor is exceeded. Each pixel value y of the conventional method is recalculated using the maximally saturated color values c low , c high calculated form each pixel value x low , x high to suppress the hue distorition.Therefore, we propose replacing c out with c ′ out to reduce the hue distortion, as,

$$y ^ { \prime } _ { o u t } = a _ { w } w + a _ { k } k + a _ { c } c ^ { \prime } _ { o u t }, \quad \quad ( 6 ) \quad \text{meth}$$

where glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

$$y _ { \text{end} } & = \begin{cases} c _ { \text{low} } & \text{if $x_{\text{low} \neq 0$ and $x_{\text{low} \neq 1$} } \\ c _ { \text{high} } & \text{if $x_{\text{high} \neq 0$, $x_{\text{high} \neq 1$, $x_{\text{low} = 0$, and $x_{\text{high} = 1$} } \\ c _ { \text{out} } & \text{if $x_{\text{low} = 0$, $x_{\text{low} = 1$, $x_{\text{high} = 0$, and $x_{\text{high} = 1$} } \\ & \text{where} \end{cases} \quad \text{and} \quad \sim \begin{cases} 0 & 1 & \text{and} \quad \end{cases} \end{cases}$$

glyph[negationslash]

where x low and x high ∈ [0 , 1] are pixel values in the raw images, X low and X high , and the maximum saturated colors c low and c high ∈ [0 , 1] 3 are calculated from x low , x high . RGB images, Y low and Y high are first calculated by applying a demosaicing algorithm to X low and X high , and then c low and c high are calculated from pixel values y low and y high of Y low and Y high , respectively. Eq.(7) allows us to is to avoid using the pixels at which over-exposure and under-exposure occurs.

## IV. EXPERIMENT

In an experiment, the performance of the proposed scheme was compared with the conventional single-shot imaging with SVE .

## A. Dataset

564 input SVE images X were prepared by using 141 HDR images selected from a database [11]. Four SVE image sets with two exposure values ± 1 EV, ± 2 EV, ± 3 EV, or ± 4 EV were generated as X from each HDR image.

## B. Objective metrics

The hue distortion of images produced by each method was evaluated in two objective metrics; the cosine similarity of maximally saturated color values, and the difference of hue values in CIEDE2000 [12]. The difference of hue values between a reference image (HDR) and the generated one was calculated for each pixel, and then the average value of all pixels was computed. For cosine similarity, a larger value means higher quality, and for the difference of hue values, a smaller value means higher quality.

The quality of images produced by each method was evaluated in the objective metrics; the tone mapped image quality index (TMQI) [13]. TMQI measure the quality of a tone mapped image from an HDR image and it consists of structural fidelity and statistical naturalness. For TMQI, a larger value means higher quality.

## C. Experiment results

From Table I and Table II, it is confirmed that the proposed method had higher scores than conventional method. Therefore, the proposed method is effective for improving hue distortion.

From Table III, it is confirmed that the proposed method had lower scores than conventional method. Although the image quality decreases slightly, the performance of the conventional method can be maintained in terms of TMQI.

TABLE I

AVERAGE SCORES OF THE MAXIMALLY SATURATED COLOR SIMILARITY

TABLE II

|                     |   ± 1EV |   ± 2EV |   ± 3EV |   ± 4EV |
|---------------------|---------|---------|---------|---------|
| Conventional method |  0.925  |  0.929  |  0.9308 |  0.9304 |
| Proposed method     |  0.9302 |  0.9382 |  0.9432 |  0.9441 |

AVERAGE SCORES OF THE DIFFERENCE HUE VALUES IN CIEDE2000

|                     |   ± 1EV |   ± 2EV |   ± 3EV |   ± 4EV |
|---------------------|---------|---------|---------|---------|
| Conventional method |   15.15 |   14.73 |   15.07 |   15.28 |
| Proposed method     |   15    |   14.53 |   14.87 |   15.23 |

TABLE III AVERAGE SCORES OF TMQI

|                     |   ± 1EV |   ± 2EV |   ± 3EV |   ± 4EV |
|---------------------|---------|---------|---------|---------|
| Conventional method |  0.2126 |  0.212  |  0.2106 |  0.2091 |
| Proposed method     |  0.2121 |  0.2115 |  0.2101 |  0.2084 |

## V. CONCLUSION

In this paper, we proposed a novel single-shot high dynamic range imaging scheme with SVE considering hue distortion. We considered estimating the correct hue of a scene from raw images, and proposed a method with the estimated hue information for correcting the hue of SVE images on the constant hue plain in the RGB color space.

## REFERENCES

- [1] Y. Kinoshita and H. Kiya, 'Automatic exposure compensation using an image segmentation method for single-image-based multi-exposure fusion,' APSIPA Transactions on Signal and Information Processing , vol. 7, 2018.
- [2] Y. Kinoshita, S. Shiota, and H. Kiya, 'A pseudo multi-exposure fusion method using single image,' IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , vol. 101, no. 11, pp. 1806-1814, 2018.
- [3] Y. Kinoshita and H. Kiya, 'Scene Segmentation-Based Luminance Adjustment for Multi-Exposure Image Fusion,' IEEE Trans. Image Processing. , vol. 28, no. 8, pp. 4101-4116, August 2019.
- [4] A1EX, 'Dynamic range improvement for some canon dslrs by alternating iso during sensor readout,' http://acoutts.com/a1ex/dual iso.pdf, 2013.
- [5] H. Cho and S. Lee, 'Single-shot high dynamic range imaging using coded electronic shutter,' in Computer Graphics Forum , vol. 33, July 2013.
- [6] S. Hajsharif, J. Kronander, and J. Unger, 'HDR reconstruction for alternating gain (ISO) sensor readout,' April 2014.
- [7] R. Gil Rodr´ ıguez and M. Bertalm´ ıo, 'High quality video in high dynamic range scenes from interlaced dual-iso footage,' The Society for Imaging Science and Technology (IS&amp;T), 2016.
- [8] U. Yoshiaki, M. Hideaki, K. Takanori, and E. U. Noriaki Suetake, 'Huepreserving color contrast enhancement method without gamut problem by using histogram specification,' in Proceeding ICIP . IEEE, 2018, pp. 1123-1127.
- [9] H. Kobayashi and H. Kiya, 'Jpeg xt image compression with hue compensation for two-layer hdr coding,' arXiv preprint arXiv:1904.11315 , 2019.
- [10] A. Visavakitcharoen, Y. Kinoshita, and H. Kiya, 'Pure-color preserving multi-exposure image fusion,' in International Workshop on Advanced Image Technology (IWAIT) 2019 , vol. 11049. International Society for Optics and Photonics, 2019, p. 110493X.
- [11] hdrlabs, 'sIBL Archive,' http://www.hdrlabs.com/sibl/archive.html.
- [12] M. R. Luo, G. Cui, and B. Rigg, 'The development of the cie 2000 colour-difference formula: Ciede2000,' Color Research &amp; Application , vol. 26, no. 5, pp. 340-350, 2001.
- [13] H. Yeganeh and Z. Wang, 'Objective quality assessment of tone-mapped images,' IEEE Transactions on Image Processing , vol. 22, no. 2, pp. 657-667, 2013.
Chihiro Go Yuma Kinoshita Sayaka Shiota Hitoshi Kiya
Single-Shot High Dynamic Range Imaging with Spatially Varying Exposures Considering Hue Distortion
We proposes a novel single-shot high dynamic range imaging scheme with
spatially varying exposures (SVE) considering hue distortion. Single-shot
imaging with SVE enables us to capture multi-exposure images from a single-shot
image, so high dynamic range images can be produced without ghost artifacts.
However, SVE images have some pixels at which a range supported by camera
sensors is exceeded. Therefore, generated images have some color distortion, so
that conventional imaging with SVE has never considered the influence of this
range limitation. To overcome this issue, we consider estimating the correct
hue of a scene from raw images, and propose a method with the estimated hue
information for correcting the hue of SVE images on the constant hue plain in
the RGB color space.