1. Field
Embodiments of the present invention generally relate to super-resolution imaging, and more particularly, to a method and apparatus for forming super-resolution images from RAW data representative of color-filter-array (“CFA”) images.
2. Related Art
Today's image capturing devices, such as digital cameras, camcorders, mobile phones, smart phones, Personal Digital Assistants (“PDA”) and any other like-type device, each include an imaging module that can be used to capture a number or sequence of images (each a “captured image”). To facilitate this, the imaging module includes a sensor, which in turn, includes an array of elements or pixels (collectively “pixels”); and generally a color filter array (“CFA”) overlaying the array of pixels.
Typically, the CFA includes pluralities of red, green and blue (“RGB”) optical filters; one for each pixel in the array of pixels. The CFA is generally arranged in a mosaic pattern (“CFA mosaic pattern”) in which each of the RGB optical filters is not the same color as the RGB optical filters that are vertically or horizontally adjacent. The CFA-mosaic pattern may be defined, for example, so that all odd (i.e., first, third, fifth, etc.) rows of the pixels in the array are overlaid by an alternating pattern of green and red optical filters (“alternating-GR filters”), and all even (i.e., second, fourth, sixth, etc.) row of the pixels in the array are overlaid by an alternating pattern of blue and green optical filters (“alternating-BG filters”).
The array of pixels are adapted to (i) sense, through the CFA, energy corresponding to luminescence associated with a scene (i.e., the captured image), (ii) convert such energy into respective electrical charges having electrical potentials corresponding to the luminescence, and (iii) store such charges (“stored charges”). Through scaling, the sensor makes the stored charges represent respective bytes of digital data; which have respective values proportional to the luminescence sensed by such pixels. The bytes of digital data that correspond to the entire captured image are commonly referred to as RAW and/or CFA (collectively “RAW”) data.
The sensor typically outputs the RAW data responsive to a control signal, such as a signal to capture another image. This RAW data, as such, represents the CFA mosaic pattern. As a result, the RAW data does not represent a full color image. This is because the RAW data does not include data for all RGB planes at each of the pixels. That is, the RAW data is missing data for (i) red and blue channels at the pixels overlaid with green optical filters; (ii) red and green channels as at the pixels overlaid with blue optical filters; and (iii) and green and blue channels at pixels overlaid with red optical filters.
Legacy image-processing devices (which may be part of or completely separate from the aforementioned image capturing devices) use demosaicing to estimate missing color channels based on the RAW data. To carry out demosaicing, a legacy image-processing device typically applies, for each of the color planes, a bicubic interpolation to the RAW data. This causes the legacy image-processing device to interpolate each of the pixels using color channel information in the RAW data for pixels neighboring such pixel.
However, an image rendered from demosaicing (“demosaiced image”) usually has colors not originally present in scene captured by the sensor (“false colors”). To reduce false colors, the legacy image-processing device may filter the already demosaiced image using one or more low-pass filters. Unfortunately, such filtering typically causes the demosaiced image to suffer from a blurring effect and other artifacts.
In addition, the interpolation will not produce the demosaiced image or will produce the demosaiced image with poor quality when the captured image is under sampled. Further, the interpolation does not restore any high frequency components and generally cannot account for different noise levels (e.g., different ISO levels). Moreover, the interpolation typically causes the demosaiced image to have an amount of pixels or resolution (“demosaiced-image resolution”) lower than the captured image. At best, the demosaiced-image resolution is the same as the captured image; the difference between being that each of pixels of the demosaiced image may have contributions from each of the RGB planes.
Often, requirements for displaying, printing and/or further processing, necessitate an images having resolutions higher than that provided by the demosaiced image. To overcome this, the legacy image-processing devices have employed an imaging processing technique commonly referred to as “Super Resolution.”
Super Resolution differs from interpolation in that it can restore to the captured image the high frequency components that correspond to finer details present in the scene, and produce an image having a resolution greater than that of the captured image (“super-resolution image”). To do this, a legacy image-processing device obtains several demosaiced images or otherwise interpolated images of the same scene; each of which is shifted, rotated or otherwise offset from the other. Thereafter, the legacy image-processing device samples such demosaiced or otherwise interpolated images to form one or more “images having resolutions lower than the demosaiced or otherwise interpolated images (“down-sampled images”). From the down-sampled images, the legacy image-processing device may form the super-resolution image. However, such super-resolution image usually has artifacts and blurring effects the same as or similar to those present in any of the demosaiced or otherwise interpolated images.