Wide-angle lenses are commonly used in cameras amongst other image and video capture devices where a large viewing angle is required or preferred. There also exist so-called Ultra Wide-Angle (UWA) lenses, with Field of Views (FOV) of up to or even larger than 180°. A fisheye lens is an exemplary UWA that may cover up to 180° (2π steradians) of angular FOV. As such, having two fisheye lenses sitting back to back and facing opposite directions will in principle cover a 360° view of the surroundings.
Image capture systems equipped with UWA lenses have many useful applications. Examples are surveillance and security monitoring cameras, video-conferencing systems, specialty photographic cameras, and vehicle camera systems. On a negative side, these lenses exhibit significant amount of distortions that pose a number of challenges. For example, the output of a typical fisheye lens is normally a two-dimensional (2D) circular (oval for a general wide-angle lens) mapping of a 3D visually familiar Euclidian space where objects lie in real world. Inevitably, horizontal and vertical lines would look skewed and appear curvilinear, more noticeably for the objects farther away from the center of the lens. The relative sizes of objects closer to the lens look exaggeratingly larger, so that different parts of an object look out of proportion. These effects are generally known as the fisheye distortions that make it difficult to judge distances and object relationships. This is in contrast with standard camera lenses which maintain the scene perspective and linearity by capturing a planar view. Moreover, to achieve a larger angular FOV, manufactures may use a combination of several lenses. That may further introduce optical and geometric distortions.
Several techniques are known in the art to address the fisheye distortions. The curvilinear images produced by a fisheye lens can be remapped onto a conventional rectilinear display. In a series of prior art patents from OmniView and IPIX Corp. a portion of the full viewing sphere is selected and corrected for fisheye distortions. Although this approach provides a powerful tool for viewing up to 360°, only a selected portion is available at a time. The rest of the image is saved that could be available upon further selections.
U.S. Pat. No. 7,961,980B2 utilizes cylindrical projections in connection with circular fisheye mappings. As is well known, cylindrical projections stretch distances east-west. In particular, the stretching diverges to infinity at the poles. This solution would be useful for creating panoramic images when the polar areas can be chopped off, again meaning loss of information. Although it is possible to take different orientation cylinders to build several images, still this does not provide a single zero-content-loss (ZCL) image, that is, an image containing all content from the original UWA captured scene, with no loss of information. In fact, it is physically impossible to achieve full perspective correction for all portions of the image at the same time by applying any transformation.
Therefore, the main drawback of prior art is loss of image content and information pertaining to objects and their relative positioning. In certain applications, such as a security or monitoring camera, it would be desired to be able to select areas of interest and correct for distortions, and yet be able to preserve the full content of the image at all times. The present invention discloses an intermediate approach, whereby image data for all pixels of the entire UWA lens captured image is maintained to prevent any loss of information after the mapping that corrects for distortions to a certain level, as much as theoretically allowed. Furthermore, a locally adaptive transformation can be applied to the regions of interest that aims to partially restore the distortions caused by the UWA lens. Partial image restoration has the objective of giving a reasonable indication of the object and its relationships to the surroundings, not a strict full-image perspective correction. This solution will be particularly useful for video monitoring applications where a full view of the surveillance must be available for every image frame, yet it would be possible to transform objects of interest.