1. The Field of the Invention
The present invention relates to an apparatus and method for determining whether or not fog is present in the environment around a vehicle based on an image which is captured by a vehicle-mounted imaging device such as a camera. For example, the present invention relates to a system and method for determining whether or not an image captured by a vehicle-mounted imaging device is hazed by the presence of fog in an environment around a moving vehicle using an image processing technique in which an high-luminance obstacle is masked.
2. Description of the Related Art
Operators in automotive vehicles in foggy regions encounter hazardous poor visibility conditions. In normal circumstances, an operator in an automotive vehicle gains knowledge of the road alignment through the configuration of markings or the landscape ahead. However, poor visibility conditions often lead to traffic accidents causing human fatalities and property damage. Although the accuracy of weather forecasting has steadily improved, accurately predicting visibility conditions is considered extremely difficult. One of the difficulties lies in that poor visibility frequently occurs in localized areas as a result of microclimate changes in that region, and the condition change in a matter of minutes or seconds. With the aim of reducing the number of the traffic accidents, automotive vehicles are increasingly being mounted with active safety systems. Many of the active safety systems installed in the automotive vehicles include perception sensors such as cameras, laser, radar and the like.
Information obtained by these perception sensors is utilized to perform automatic tasks such as turning on fog lamps or alerting the operator that the safety system installed in his vehicle is inoperative due to lack of sufficient information from the sensors. Hence, in a certain sense, information obtained by a system or a block capable of detecting the presence of fog is a fundamental one for driving assistance. Indeed, in a foggy environment, the operator actually tends to overestimate visibility distance and to drive his vehicle with excessive speed. Therefore, it is important to detect the presence of fog around the vehicle.
The fog effects on the atmospheric visibility are modeled by Koschmieder's law on the apparent luminance of observed objects against background sky on the horizon. In Koschmieder's law, one of the parameters is the extinction coefficient k of fog. In fog, a proportion of the light is scattered by water droplet. Because the absorption of visible light by water droplets can be negligible, the scattering and extinction coefficient are considered to be interchangeable. Koschmieder's law states a simple relationship between the distance d of an object with intrinsic luminance L0 and its apparent luminance L as follows:L=L0e−kd+L∞(1−e−kd),where L∞ denotes the luminance of the atmosphere, such as the luminance of the sky, and k denotes the extinction coefficient of the atmosphere. This expression indicates that the luminance of the object observed through fog is attenuated as e−kd, and a luminance reinforced by daylight scattered from the atmospheric particles between the object and the observer has a form of L∞(1−e−kd).
In addition to luminance, contrast can be defined by the following equation:
      contrast    =                            L          0                -                  L          ∞                            L        ∞              ,where L0 and L∞ have the same meaning in the equation of Koschmieder's law. When the object is darker than its background, that is L0 is less than L∞, contrast C is negative. By the definition of contrast, an attenuation law of atmospheric contrast can be derived as follows:C=C0e−kd,where C is the apparent contrast at distance d and C0 is the intrinsic contrast of the object against its background.
In practice, the fact that the object is visible is identified when the value of the apparent contrast C is greater than or equal to a threshold value Cth. In general, the visual range V is defined as the greatest distance at which a black object (C0=1) can be seen in the sky on the horizon with the threshold value Cth=0.05:
  V  =                    -                  1          k                    ⁢              ln        ⁡                  (          0.05          )                      ≅                  3        k            .      
The above mentioned theory has been used to determine whether or not fog is present in the outside atmosphere, surrounding the field of view from the operator's seat in a vehicle using an image captured by a vehicle-mounted camera, although Koschmieder's law is derived assuming that atmospheric illumination is uniform.
For example, in WO 03/069275, Lavenant et al. disclose a method for determining whether or not the environment of a vehicle is foggy. The method of Lavenant et al. includes at least four steps for determining the presence of fog: the first step is recording at least one first image of the environment in which a vehicle is traveling, from the vehicle in the direction of travel of the vehicle, the second step is a step of recording a luminance at each point of the first image, the third step is a step of searching a region within the first image that displays minimal line-to-line gradient variation when crossed from bottom to top, in a configuration that allows for compatibility with Koschmieder's law, that is a vertical luminance curve along the vertical axis of the region has at least one point of inflection, and the fourth step is a step of calculating a coefficient of extinction for the fog from the vertical luminance curve. If such vertical luminance curve is found, the presence of fog in the environment of the vehicle is detected. It should be noted that in the method of Lavenant et al., the third step includes steps of estimating the similarity of a pixel to the one located just below through use of a filter that applies some different masks to the first image and computes an average and dispersion of the luminance for the masked images to feature the smallest level of dispersion.
However, in the method of Lavenant et al., there is a need for identifying a region of sky within an image captured by a vehicle-mounted camera because the apparent luminance of an object against its background is needed in order to apply Koschmieder's law.
Further, Schechner et al. disclose in WO 2007/083307 a system and method for estimating and correcting outdoor images captured by a camera and plagued by poor visibility conditions due to atmospheric scattering, particularly haze, implicitly using Koschmieder's law. In order to correct the images caused by poor visibility conditions, subtraction of airlight and correction for atmospheric attenuation by haze should be performed. In the method of Schechner et al., airlight and attenuation parameters are computed by analyzing polarization-filtered images, in particular without identifying sky areas within the images. That is, in conventional methods, these parameters were estimated by measuring pixels in sky areas. Because the method of Schechner et al. uses the fact that the airlight is often partially polarized in haze, the method of Schechner et al. is considered to be applicable for estimating these parameters and for determining the presence of fog, for example, the presence of dilute fog, from captured images by a camera, when the sky is not in view. However, the method of Schechner et al. needs to use at least two images at different polarization states.
Schechner et al. assume that the camera for capturing the images is not moving, but is fixed at some position. Hence, if it is intended that the system of Schechner et al. is installed in a vehicle, a plurality of vehicle-mounted cameras should be needed to capture simultaneously at least two images. This fact lead to a more complex system when the method of Schechner et al. applies to a vehicle-mounted fog detecting system which is not suitable for practical purposes.
Further, the method of Schechner et al. may be less effective when illumination is less directional. The degree of polarization of airtight is decreased by depolarization which caused by multiple scattering, as occurs in fog. Moreover, in general, haze is constituted of aerosol which is composed of molecules having radius of 10−2˜100 μm, in contrast to fog that is water droplet having radius of 100˜101 μm. Reduction of polarization is also caused by scattering from large particles, even scattering from large haze particles. Therefore, for example in dense fog, an accurate determination of the presence of fog is difficult.
Further, Leleve et al. disclose, in French patent publication No. 2,847,367, a method and system for determining the range of visibility for an operator in a vehicle in the presence of an element disrupting the visibility of the operator and for determining the presence of fog. The method of Leleve et al. includes the following six steps. The first step is a step of capturing at least one image of a field of space located in front of the vehicle, wherein the image is defined by an array of pixels and sweep-lines. The second step is a step of separating the image in two parts by a first vertical line passing through a pre-set point. The third step is a step of determining the luminosity of the pixels of the first vertical line to obtain a curve of luminosity. The fourth step is a step of determining a first tangent to the curve of luminosity tangential at a place of the curve representative of a region of luminosity substantially independent of the disruptive element, such as in the sky. The fifth step is a step of determining a second tangent parallel to the first tangent to the curve of luminosity tangential at a place of the curve representative of stabilization of the luminosity, such as on the road. The sixth step is a step of determining a sweep-line according to the first tangent and the second tangent, wherein the sweep-line is representative of the distance of visibility. In the second step mentioned above, the pre-set point is determined by the following steps: a step of searching zones of the image, each zone responding to the predicate of homogeneity, determining the center of gravity for each of the zones, and determining the global center of gravity of each of centers of gravity for the zones, the global center being the pre-set point. Thus, the global center of gravity is the resultant of centers of gravity for two homogeneous zones on the road and in the sky. The sweep-line is calculated according to an intersection point between the curve of luminosity and a line parallel to the first tangent at a middle point between the first tangent and the second tangent. The range of visibility is determined according to the sweep-line.
That is, the method of Leleve et al. is based on the search for a vertical light gradient. Leleve et al. considered that the disruptive element is fog. As explained above, the method of Leleve et al. is based on the search for a homogeneous region in an image of the road scene and on the search for gradient of vertical light. These searches make it possible to establish a relationship between the range of visibility for the operator in the vehicle and the reversal point (the sweep-line) in the image and to determine whether or not fog is present.
The method of Leleve et al. is applicable to the case where there is an obstacle on the road over which the vehicle is traveling, for example, a bridge and the like. However, their solution presents the following disadvantages. First, in order to perform the method, it is necessary that the image taken by the camera with which a vehicle equipped contains a homogeneous zone in the sky. Second, if there is another vehicle in front of the vehicle and the other vehicle is emitting light from the tail lamp, it is difficult to determine the curve of luminosity.
There has been known another apparatus for executing an image processing on an acquired image from a vehicle-mounted camera for determining the presence of fog, as disclosed by Akutagawa in Japanese Patent No. 3444192. In Japanese Patent No. 344419, the presence of fog is determined based on the degree of image blurring. With fog being present, the image gets blurred according to Koschmieder's law. Thus, the presence of fog is determined upon the estimated degree of image blurring. In estimating the degree of image blurring, first, a differential calculus is executed for each pixel of the entire image to calculate a rate of change of edge intensities in each pixel of the entire image. The edge intensities at each of the pixels in the entire image are used to estimate the degree of image blurring.
With the apparatus disclosed in Japanese Patent No. 3444192, the result of determining the presence of fog is utilized in recognizing a white line on a road. In recognizing the white line on the road on which a vehicle is traveling based on an image captured by the vehicle-mounted camera, if fog is present in an atmosphere on the road, the image gets blurred with a resultant difficulty of recognizing the white line. Therefore, the presence of fog is determined prior to recognizing the white line. It is also disclosed that if the presence of fog is determined, fog lamps are switched on.
The apparatus of Akutagawa installed in a vehicle includes means for capturing an image on the road over which the vehicle is traveling, means for detecting a white line on the road from the image, means for calculating distances from the vehicle to each points of the white line, means for determining degree of blurring at each point of the white line, means for obtaining a curve of degree of blurring at each points of the white line as a function of the distance from the vehicle, means for comparing the curve of degree of blurring at each points of the white line with a given curve that is obtained when visibility of the operator in the vehicle is normal, and means for determining how bad visibility for the operator is.
There is a further known method for identifying poor visibility under adverse weather conditions by processing digital images, as disclosed by Hagiwara in Japanese unexamined patent application No. 2006-221467. In the method of Hagiwara, the weighted intensity of power spectra for an image is calculated as a value for identifying poor visibility. The magnitude of the weighted intensity of power spectra represents the difference in spatial frequencies within the image based on human contrast sensitivity. The weighted intensity of power spectra is calculated using a frequency analyzing technique in which the image is broken down into sinusoidal gratings of different spatial frequencies by Fourier Transform. From the Fourier transform, the power spectrum value corresponds to the amplitude of spatial frequency. Each component of the spatial frequencies of the image shows a corresponding sinusoidal diffraction grating indicating patterns of grayscales in the image. The power spectrum of the spatial frequency indicates amplitudes of each sinusoidal diffraction grating. As visibility in the road decreases in daytime in fog, the number of sinusoidal diffraction gratings of high spatial frequency in the road image decreases and the amplitude of all sinusoidal diffraction grating in the road image becomes small.
The method of Hagiwara includes steps of: an inputting step of inputting a digital image, a Fourier transforming step of performing Fourier transform on the digital image to obtaining the corresponding power spectrum of each component of spatial frequencies within the digital image, a filtering step of filtering predetermined components of spatial frequencies based on human contrast sensitivity function so as to obtain a distribution of intensities of the power spectrum of a filtered digital image, and a determining step of determining of whether or not poor visibility is realized or whether or not fog is present according to the distribution of the intensities of power spectrum. In this method, even if visual angle or resolution of an image capturing block such as a camera or a charge-coupled block (CCD) are changed, it is possible accurately of determine whether or not fog is present. Therefore, there is an advantage in the method in which spatial frequency analyzing technique is utilized over the method for determining whether or not fog is present based on the degree of image blurring that is computed from the rate of change of edge intensities of each pixel of the entire image.
However, if it is intended to determine whether or not fog is present either using the apparatus of Akutagawa in which a degree of image blurring is calculated or using the method of Hagiwara in which spatial frequency analysis technique is utilized, the effects of obstacles contained in a frame of the image capturing block must be eliminated in order to expel noise components from the image to be examined in the edge intensity analysis or the spatial frequency analysis by masking the obstacles within the image. The obstacles generating noise in the edge intensity analysis or the spatial frequency analysis include a lamp standing beside the road, a preceding vehicle in front of the vehicle whose image is not substantially blurred, and a pole beside the road such as an utility pole, and the like. The pictures of these obstacles can be clearly recognized even if in fog, and they need to be masked.
However, if masked portion of the image is increased, it becomes difficult to perform an accurate calculation in the edge intensity analysis or the spatial frequency analysis due to lack of sufficient information in the masked image. In such case, accuracy in determining whether or not fog is present can not be ensured.