This section is intended to provide a background or context to this invention. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.
Various abbreviations that appear in the specification and/or in the drawing figures are defined as follows:
CFAI color filter array interpolation
DPCM differential pulse-code modulation
DSP digital signal processor
FPGA field-programmable gate array
GPU graphics processing unit
HW hardware
ISP image signal processor
LUT look-up table
OTF on-the-fly
PCM pulse-code modulation
ROI region of interest
SIMD single instruction, multiple data
SW software
As shown in FIG. 1, a typical camera system 100 contains at least one camera sensor 110, an image processing engine (also called an ISP) 120, which may include an image buffer 125, a compression engine 130 and memory 140 for storing an image file. The camera system 100 may also include a display 150. The camera sensor 110 can capture an image that contains x pixels in the horizontal direction and y pixels in the vertical direction. The image is read from the camera sensor 110 to the image buffer 125. From the image buffer 125, the image can be processed by the ISP 120, compressed by compression unit 130 and then stored in memory 140. Additionally, the image may also be displayed on the display 150 either directly from the ISP 120 or from memory 140.
It is also possible to process images on the fly (OTF), where the image is read from the camera sensor 110 and processed directly without buffering.
Generally, image processing is done either in a software ISP (SW ISP) or in a hardware ISP (HW ISP). Both methods have their own related problems that affect image quality and resources needed from the device. It may also be possible to process images using the HW processing and using SW processing (before or after the HW ISP) for some enhancements or additional processing to the same image pixels.
When an image is processed using only HW processing, the image is typically cropped. As the size of hardware is minimized to conserve space and costs, the processing is also simplified as much as possible. Thus, complex processing (e.g., the processing required for lines or columns in border areas) is simplified and some pixels may not be used for the final output image. Although the camera sensor 110 may contain a large number of pixels (e.g., 5.1 Million pixels), the image file will contain less pixels (e.g., 4.9 Mpix out of the possible 5.1 Million pixels).
Another problem is that typically the quality of the HW processing is not as good as the quality of the SW processing.
When an image is processed using only a SW ISP the processing may be too slow and cause additional delays (latencies, e.g., shot-to-save increases). The problem is more severe if there is no background processing solution used to minimize those latencies (thus shot-to-shot latency also increases).
One solution is to use a larger HW ISP that can process border areas and other pixels properly. However, this may increase both the size and cost of the camera.
Some prior solutions try to combine SW and HW image processing.
The system proposed in JP2006173843 is one such the system. The process splits an image into bands (or slices) that can be processed within a SW ISP or a HW ISP based on the availability of the processing systems. It divides the image into the predetermined slices, but it does not specify which slice is processed with a HW ISP and which slice is processed with a SW ISP except to enhance execution efficiency (and thus reduce processing delays).
Another proposed system is described in JP2006085414. This system is used for image rendering. It utilizes SW processing when the processing requirements exceed the limited HW capabilities. For example, if the image is requested to be zoomed 10 times and the HW is capable of zooming to only 4 times, then the image is temporarily zoomed at least 2.5 times by SW processing and the remainder of the zooming is performed with hardware. Thus, the same image area is processed with a HW ISP and with a SW ISP similarly to other SW enhancement procedures.
Another approach is to use the same HW ISP two (or more) times. This would require a buffering mechanism during processing. A buffering mechanism may also be needed within the combined SW solutions, as typically a SW ISP is not capable of performing OTF processing at the same speed as a HW ISP.