Conventional cameras are excellent for capturing realism in scenes. However, conventional images are frequently insufficient for many applications where instead of physical realism, altered images are desired. The alteration can enhance some details, while de-emphasizing other details.
Therefore, many modern digital cameras perform image processing, such as non-linear color mapping, edge sharpening, or saturating blue shades in images of outdoor scenes to make water or sky look more lively, while still maintaining near-photorealistic appearance.
Similarly, a photographer can illuminate a scene with soft, side or back lighting to selectively add contrast to the scene. In addition, images can be retouched using a photo-editor that applies various image filters and enhancers.
Besides being aesthetically pleasing, enhanced and reduced complexity images are also useful for highlighting or clarifying selected features in technical illustrations, or high-quality thumbnail images. With a few exceptions, available techniques for image enhancement, image abstraction or image stylization involve capturing and processing a single image as the input, see DeCarlo et al., “Stylization and Abstraction of Photographs,” Proceedings of Siggraph '02, ACM Press, 2002, and Hertzmann, “Painterly Rendering with Curved Brush Strokes of Multiple Sizes,” Proceedings of Siggraph '98, ACM Press, 1998.
Even with state-of-the-art passive and active image processing techniques, it is still difficult to acquire accurate depth, normal and shape information. It is also difficult to generate novel views of a scene from acquired images.
In computer graphics, one approach hides these problems by using high quality images. Texture mapping can ‘hide’ low quality in range images, and light fields enable data-dense views instead of relying on data representation.
At the same time, digital cameras continue to evolve at a rapid pace. Advancements include higher pixel resolutions, more bits per pixel to increase the dynamic range, higher frame rates, and the ability to take multiple pictures with different lengths of exposure and wavelength, e.g., infrared.
Non-photorealistic images (NPR) intentionally appear different from photographs. NPR images can be classified broadly as artistic or utilitarian, e.g., technical illustrations. NPR images can emphasize important features such as object edges. Moving parts can be shown in different colors, and less important details such as shadows and other ‘clutter’ can be reduced or eliminated. Furthermore, NPR images can decouple image resolution from contained information.
The input for stylized image generation can be 3D geometric representations or images, see Markosian et al., “Real-Time Nonphotorealistic Rendering,” Proceedings of Siggraph '97, Whitted, Ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, pp. 415–420, 1997, Gooch et al., “Using Non-Photorealistic Rendering to Communicate Shape,” Siggraph '99 Course Notes, Course on Non-Photorealistic Rendering, Green, Ed. Ch. 8., 1999, Hertzmann, “Introduction to 3D Non-Photorealistic Rendering: Silhouettes and Outlines,” Siggraph '99 Course Notes on Non-Photorealistic Rendering, Green, Ed. New York, Ch. 7, 1999, and Kowalski, “Art-Based Rendering of Fur, Grass, and Trees,” Proceedings of Siggraph '99, Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, pp. 433–438, 1999.
Prior art techniques for generating stylized images from a single image have involved morphological operations, image segmentation, edge detection and color assignment. However, those techniques are limited by their dependence on a single input image. Some of techniques aim for stylized depiction, see Ostromoukhov, “Digital facial engraving,” Proceedings of Siggraph '99, Rockwood, Ed., ACM SIGGRAPH, pp. 417–424, 1999, while others try to enhance legibility. Interactive techniques such as rotoscoping are effective as well.
Methods to combine information from multiple images into one have been explored for various other applications. They vary from tone mapping for compression of variable-exposure high-dynamic range images, see Fattal et al., “Gradient Domain High Dynamic Range Compression,” Proceedings of Sigggraph '02, ACM SIGGRAPH, 2002, and Reinhard et al., “Photographic Tone Reproduction for Images,” Proceedings of Siggraph '02, ACM SIGGRAPH, 2002.
Some techniques consider changing atmospheric conditions to extract 3D information and perform fog elimination, see Nayar et al., “High dynamic range imaging: Spatially varying pixel exposures,” IEEE CVPR, 2000. Active illumination methods have been used for depth extraction and photometric stereo. Unfortunately, active illumination is unstable at depth discontinuities, which are critical for stylized rendering.
Helmholtz stereopsis attempts to overcome some of these problems, see Zickler, “Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction,” ECCV, 2002. Other active methods such as shadow carving compute a tighter hull by observing shadows, see Savarese et al., “Shadow Carving,” Proc. of the Int. Conf. on Computer Vision, 2001.
Therefore, it is desired to provide a camera and an image rendering method that can use multiple images acquired of a scene under different illumination conditions to generate an output stylized image with enhanced or de-emphasized information.