Image sensors are semiconductor devices that capture and process light into electronic signals for forming still images or video. Their use has become prevalent in a variety of consumer, industrial, and scientific applications, including digital cameras and camcorders, hand-held mobile devices, webcams, medical applications, automotive applications, games and toys, security and surveillance, pattern recognition, and automated inspection, among others. The technology used to manufacture image sensors has continued to advance at a rapid pace.
There are two main types of image sensors available today: Charge-Coupled Device (“CCD”) sensors and Complementary Metal Oxide Semiconductor (“CMOS”) sensors. In either type of image sensor, a light gathering photosite is formed on a semiconductor substrate and arranged in a two-dimensional array. The photosites, generally referred to as picture elements or “pixels,” convert the incoming light into an electrical charge. The number, size, and spacing of the pixels determine the resolution of the images generated by the sensor.
Modern image sensors typically contain millions of pixels in the pixel array to provide high-resolution images. To capture color images, each pixel is covered with a color filter, an optical element that only allows penetration of a particular color within given wavelengths of light. A color filter array (“CFA”) is built on top of the pixel array for separating color information for each pixel. The most popular type of CFA is called a “Bayer array”, composed of alternating rows of Red-Green and Green-Blue filters. The Bayer array has twice as many Green filters as Blue or Red filters to account for the Human Visual System peak sensitivity to the green portion of the light spectrum. The image information captured in each pixel, e.g., raw pixel data in the Red, Green, and Blue (“RGB”) color space, is transmitted to an Image Signal Processor (“ISP”) or other Digital Signal Processor (“DSP”) where it is processed to generate a digital image.
The quality of the digital images generated by an image sensor device depends mostly on its sensitivity and a host of other factors, such as lens-related factors (distortion, flare, chromatic aberration, depth of field), signal processing factors, time and motion factors, semiconductor-related factors (dark currents, blooming, and pixel defects), and system control-related factors (focusing and exposure error, white balance error). In particular, lens distortion may significantly affect the quality of the digital images.
Lens distortion is an optical aberration that occurs when the object being imaged is not situated on the optical axis of the lens. The result is an error between the digital image coordinates and the object coordinates. Among various types of lens distortions, radial distortion along a radial line from the optical center of the digital image is one of the most common and severe.
Digital images with radial distortion have distorted lines that are bent away and/or toward the center of the image, as illustrated in FIGS. 1A-B. In FIG. 1A, the distorted image 105 appears to have a spherical or barrel shape with lines bent away from the center of the image as image magnification decreases with increasing distance from the optical center. This type of radial distortion is commonly referred to as “barrel” distortion. Barrel distortion is associated with wide-angle lenses (e.g., fisheye lenses) and typically occurs at the wide end of a zoom lens. Wide-angle lenses have become popular in many image sensor devices and applications, such as, for example, in automobile back-up cameras and in medical devices.
The distorted image 115 shown in FIG. 1B has the opposite effect, with lines bent toward the center of the image as image magnification increases with increasing distance from the optical center. This image is said to have “pincushion” distortion as its lines appear to be pinched at the center. Pincushion distortion is associated with telephoto lenses and typically occurs at the telescope end of a zoom lens.
Another type of radial distortion is moustache distortion, which is a mixture of barrel distortion (at the center of the image) and pincushion distortion (on the periphery of the image). Moustache distortion may be observed on retrofocus lenses or large-angle zoom lenses.
All three of these radial distortion types can introduce significant errors in the digital images and must be corrected for. Existing approaches to correct for lens distortion in digital images range from post-processing techniques applied to the digital images themselves to techniques used in the image sensor devices before the digital images are generated. Post-processing techniques are widely available in software packages and solutions such as Adobe® Photoshop®, developed by Adobe® Systems Inc., of San Jose, Calif. However, they are not suitable for real-time applications requiring distortion-free or close to distortion-free images such as medical, surveillance, and navigation applications.
In this case, processing techniques can be used in the image sensor devices at capture time before the digital images are generated. These techniques include the use of distortion models that estimate the distortion in the digital images based on model parameters that are derived during lens calibration. The calibration process uses a known test pattern or image to determine the extent of the distortion introduced by the lens. The distortion is modeled with a mathematical function, such as a nth-order polynomial, that is then applied to the distorted image to correct its coordinates.
For example, let the coordinate of a given pixel in a distorted image be (x,y). The pixel (x,y) is mapped to a coordinate (x′,y′) in a corrected image according to a mathematical function ƒ, that is, (x′,y′)=ƒ(x,y,c), where c denotes distortion parameters determined during lens calibration. Typically, both horizontal and vertical coordinates are mapped together, that is, the model corrects for both horizontal and vertical lens distortion simultaneously. A given set of distortion parameters c may be stored in the image sensor device and applied for a number of lenses within a lens family. The goal is for the corrected image to be as distortion free as possible.
In practice, applying a distortion model in an image sensor device may require a large number of line buffers to store the coordinates of distorted lines before the model is applied. The line buffers are small, temporary storage locations used to store a line or part of a line of image data. Depending on the lens, a large number of line buffers may be required to store neighboring lines before a new coordinate in a corrected image may be determined by the lens' distortion model. For example, the Sunex DSL215 lens, developed by Sunex, Inc., of Carlsbad, Calif., may require around 70 line buffers to apply its distortion model and generate corrected images. Such requirements increase power consumption, die size, and system cost of image sensor devices.
Accordingly, it would be desirable to provide an image sensor apparatus and method for correcting lens distortion in digital images at capture time without requiring a large number of line buffers.