The present invention relates to characterizing colors in an image, as well as to measuring colors in an image and matching colors between different image objects in a template and a target image.
In machine vision applications, color is a powerful descriptor that often simplifies object identification and extraction from a scene. Color characterization, location, and comparison is an important part of machine vision and is used in a large class of assembly and packaging inspection applications. Inspection involves verifying that the correct components are present in the correct locations. For example, color may be used to inspect printed circuit boards containing a variety of components; including diodes, resistors, integrated circuits, and capacitors. These components are usually placed on a circuit board using automatic equipment, and a machine vision system is useful to verify that all components have been placed in the appropriate positions. A method of effectively characterizing and locating components on a circuit board is desired.
Color is widely used in the automotive industry to verify the presence of correct components in automotive assemblies. Components in these assemblies are very often multicolored. For examples, color characterization may be used to characterize and inspect fuses in junction boxes, i.e. determining if all fuses are present and in the correct locations. In addition, it is often necessary to match a fabric in one part of a multi-color automobile interior. A method of characterizing a multi-colored area for analysis and comparison is desirable. A color characterization may be used to determine which of several fabrics is being used as well as the orientation (texture) of the fabric. In a manufacturing environment, color matching techniques may also be used to find flaws in a manufactured product, assuming the flaws are accompanied by a color change.
A direct measurement of the color distribution in an object may also be useful in machine vision applications. Instead of matching colors between the target image and the template image, a quantitative representation of the color distribution is produced. This data may then be used for further manipulation of the color information in machine vision applications. For example, a color measurement machine vision system may be used to test and analyze the color components in textiles or dyes. A quantitative representation is also useful in color image segmentation and color image retrieval. For example, an object in an image may be segmented based on its color, and an image may be retrieved from an image database based on its color representation.
A color space (or color model) is a way of representing colors and their relationship to each other. A color space is essentially a 3-D coordinate system and a subspace within that system where each color is represented by a single point or vector. Image processing and machine vision systems use several different color spaces including RGB, HSI (or HSL) and CMY. In the RGB space, each color appears in its primary spectral components of red, green and blue. This RGB color space is based on a Cartesian coordinate system. The RGB model is represented by a 3-dimensional cube with red, green, and blue at the edges of each axis. Each point in the cube represents a color, and the coordinates of that point represents the amount of red, green and blue components present in that color. Because the red, green, and blue color components in RGB color space are highly correlated, it is difficult to characterize colors with intensity/luminance independent features.
The Hue, Saturation, Intensity (HSI) or Hue, Saturation, Luminance (HSL) color space was developed to put color in terms that are easier for humans to quantify. The hue component is color as we normally think; such as orange, green, violet, and so on (a rainbow is a way of visualizing the range of hues). Thus, hue represents the dominant color as perceived by an observer. Saturation refers to the amount or richness of color present. Saturation is measured by the amount of white light mixed with a hue. In a pure spectrum, colors are fully saturated. Colors such as pink (red and white) and lavender (purple and white) are less saturated. The intensity or light component refers to the amount of grayness present in the image.
Colors represented in HSI model space may be ideal for machine vision applications for two reasons. First, HSI includes an intensity (luminance) component separated from the color information. Second, the intimate relation between hue and saturation more closely represents how humans perceive color. It is therefore desirable to characterize colors in HSI space for color measurement and color matching.
HSI is modeled with cylindrical coordinates. One possible model comprises the double cone model, i.e., two cones placed end to end or an inverted cone below another cone (see FIG. 4). For information on the double cone model, please see xe2x80x9cA Simplified Approach to Image Processingxe2x80x9d, Randy Crane, Prentice Hall, 1997. The hue is represented as the angle theta, varying from 0 degree to 360 degree. Saturation corresponds to the radius or radial distance, varying from 0 to 1. Intensity varies along the z-axis with 0 being black and 1 being white. When S=0, the color is gray scale with intensity I and H is undefined. When S=1, the color is on the boundary of the top cone base and is fully saturated. When I=0, the color is black and therefore H is undefined.
On the assumption that the R, G and B values have been normalized to range from 0 to 1, the following equations may be used to convert from RGB color space to HSI (or HSL) color space:
I=(R+G+B)/3
      H    =                  cos                  -          1                    ⁢              {                                            1              2                        ⁡                          [                                                (                                      R                    -                    G                                    )                                +                                  (                                      R                    -                    B                                    )                                            ]                                                          [                                                                    (                                          R                      -                      G                                        )                                    2                                +                                                      (                                          R                      -                      B                                        )                                    ⁢                                      (                                          G                      -                      B                                        )                                                              ]                                      1              2                                      }                  S    =          1      -                        3                      (                          R              +              G              +              B                        )                          ⁡                  [                      min            ⁡                          (                              R                ,                G                ,                B                            )                                ]                    
The Intensity I (or Luminance L) may also be represented by the equation:
L=0.299R+0.587G+0.114B
which is a weighted sum of the RGB values.
The equation for H yields values in the interval [0 xc2x0, 180xc2x0]. If B/I greater than G/I then H is greater than 180xc2x0 and is obtained as H=360xc2x0xe2x88x92H.
Prior art in color machine vision systems use various techniques to measure and match colors. Those skilled in the art will be familiar with xe2x80x98thresholdingxe2x80x99 an image. To threshold a color image, a threshold is applied to each of the three planes that make up the image. In RGB mode, to select a particular color, one will need to know the red, green and blue values that make up the color. In RGB mode it is not possible to separate color from intensity. Therefore, a characterization algorithm such as histogram Intersection based on RGB space will be intensity sensitive. For more information on this, please see xe2x80x9cColor Indexingxe2x80x9d, Michael J. Swain, Internal Journal of Computer Vision, vol. 7:1, page 11-32, 1991.
In the HSI color space, since the color and intensity information can be separated, one usually thresholds the color image in the hue plane to identify the dominant color (hue). However, it is difficult to distinguish multiple color objects by the thresholding technique, especially when the saturation has to be taken into account. Moreover, the black and white colors are the background colors in many machine vision applications and chromaticity(i.e. hue and saturation) can not be used to represent them. Therefore, The intensity value will also have to be used to represent black and white colors in the machine vision applications.
Prior art color matching techniques commonly calculate the color difference between corresponding pixels of a target object and a template object. These prior art techniques perform pixel by pixel comparisons or subtractions between pixels of the target object and pixels of the template object. The results of these pixel by pixel comparisons may then be compiled to determine the level of color similarity between the entire target object and template object. The computation cost of pixel by pixel comparison is very expensive and is difficult to accomplish in real time. A more efficient color characterization method is desirable. More specifically, it is desirable for machine vision applications to more effectively characterize and compare the colors in an image.
U.S. Pat. No. 5,410,637 (Kern) uses fuzzy logic to establish acceptable pass/fail tolerances for production or inventory samples. The process first stores a series of training image samples which are labeled pass or fail according to visual inspections. The initial value of the tolerance is a super ellipsoid determined by the high/low value of the pass samples in the training set. A classifier template uses the super ellipsoid tolerances and ranks every sample in the training set. The process then employs fuzzy logic to obtain an optimized tolerance which minimizes the sum of ranking error between the classifier template and the visual ranks. The process essentially builds a pass/fail color classifier. This process can not be used to measure the colors quantitatively in an image or to measure the quantitative color similarities between two objects in an image or in two separated images.
U.S. Pat. No. 5,085,325 (Jones) implements a color sorting system and method. The method creates a lookup table containing a series of 0""s (accept) and 1""s (reject) based on good and bad sample images. During the sorting process, the pixel value of the input image is used to address the lookup table, the output of the lookup table is either 1 or 0. If the number of rejects (1""s) accumulated is larger than a specified number K, the input image is rejected. This color sorting method is based on a pixel by pixel comparison. A large memory is required to store the lookup table. Although a special hardware addressing approach can improve the processing speed, the cost of computation is still very high for sorting objects with complex colors.
U.S. Pat. No. 5,751,450 (Robinson) provides a method for measuring the color difference of two digital images as a single xe2x80x98distance.xe2x80x99 This xe2x80x98distancexe2x80x99 is an average of the color differences of all corresponding pixels of the two images. Similar to the Jones"" patent as described above, the cost of computation of the distance is very high. This template image has to be stored in the computer memory for on-line color matching. If the size of the template and target image are not the same, special operations for alignment or resizing the image must be done before the matching process can begin. A further drawback of this approach is that it is impossible to have scale and rotation-invariant color matching based on the xe2x80x98distancexe2x80x99 measure.
In view of the foregoing, an object of the present invention is to provide improved systems and methods for effectively and accurately characterizing color for machine vision applications.
Another object of the invention is to provide a machine vision system for measuring multiple colors in an image. This color measuring is intensity independent within a large range of intensity variation.
Still another object of the invention is to provide a machine vision system for measuring multiple colors, including black and white color, while the color measuring system is intensity independent within a large range of intensity variation.
Still another object of the invention is to provide a machine vision system for measuring multiple colors with different saturation values in an image, while the color measuring system comprises a wide range of intensity variation and is intensity independent.
Still another object of the invention is to provide a machine vision system for color matching that may quantitatively measure the color difference between two images or between two regions of interest in the same image.
Still another object of the invention is to provide a machine vision system for color matching that is not required to calculate the color difference based on pixel by pixel comparisons.
Still another object of the invention is to provide a machine vision system for color matching that is intensity independent within a large range of intensity variation.
Still another object of the invention is to provide a machine vision system for color matching that can distinguish colors with different saturation values.
Still another object of the invention is to provide a machine vision system for color matching that compensates for black and white color distribution in images.
The foregoing objects may be attained according to the present invention, which provides, in one aspect, an improved system and method for characterizing the colors in an image. More particularly, the present invention provides a system and method for effective on-line color measurement and color matching.
The color characterization method operates to characterize the colors of an image object or region of interest (ROI). The image is obtained in HSI format, or alternatively converted from another format to HSI. For example, an image may be acquired in HSI format by National Instruments color image acquisition board PCI-1411. The color characterization divides the HSI space into n color categories (also referred to as subspaces or bins), where n is the number of color categories of interest. The number of different color categories in the color space may be dependent on a desired complexity of the color characterization.
For each image object pixel, the method determines a color category for the respective pixel based on values of the respective pixel, i.e., hue, saturation and intensity values, wherein the color category is one a plurality of possible color categories or bins (or sub-spaces) in the HSI space. The number of pixels assigned to each category is then counted and normalized by the total number of pixels in the selected region of interest (or entire image), i.e., the percentage of pixels in each color category characterizes the colors of the image or ROI. The percentage of pixels in each color category may also be used as a quantitative measurement of the color distribution of the image.
In one embodiment, the method first divides the double cone HSI color into sub-spaces so that each subspace represents a color category. All the categories together include every possible color, including black and white. In the color characterization process, the method determines if the respective pixel can be categorized as either black or white based on one or more of the saturation and intensity values of the respective pixel. If the respective pixel can not be categorized as either black or white, then the method determines a color category for the respective pixel based on hue and saturation values of the respective pixel. Thus the method operates to determine a color category for the respective pixel based on hue, saturation and intensity values of the respective pixel.
As the pixels are analyzed and assigned to color categories, the method stores information in the computer regarding the number of pixels in each of the color categories. The method may then compute a percentage of the pixels in each of the color categories, wherein this percentage information characterizes colors in the image object. The characterization information may then be output to a computer screen or other device for further color analysis in machine vision applications.
The invention further includes an improved color matching method that uses the color characterization method described above. The color matching method determines similarity of colors between a template image object and a region of interest (ROI) or target image object. The color matching method first performs the above color characterization technique on each of the template image object and the ROI to determine color information regarding the template image object and the ROI. For each of the template image object and the ROI, the color information comprises information regarding a number or percentage of pixels in each of the color categories of a color space.
In one embodiment, the color characterization and/or color matching method may perform a smoothing operation (moving filtering) on the stored color information. The smoothing may comprise, for each respective color category, distributing a portion of the number or percentage of pixels in the respective color category to one or more neighboring color categories. The color matching method then generates match information based on the color information of the template image object and the color information of the ROI, wherein this match information indicates the similarity of colors between the template image object and the ROI. The match information may be generated for each respective color category by comparing the number or percentage of template image object pixels in the respective color category with the number or percentage of ROI pixels in the respective color category. For example, the absolute distance may be used to calculate the difference of the percentages for each category and these differences summed (or weighted summed) to produce a match or score value. Where the color matching is based on percentages of colors in respective color categories, rather than the number of pixels in each category, matching may be performed between two images regardless of their respective sizes and orientation, and without expensive calculations to retest for different rotations or scales of the images.