A system and method for determining image similarity. The method includes the steps of automatically providing perceptually significant features of main subject or background of a first image; automatically providing perceptually significant features of main subject or background of a second image; automatically comparing the perceptually significant features of the main subject or the background of the first image to the main subject or the background of the second image; and providing an output in response thereto. In the illustrative implementation, the features are provided by a number of belief levels, where the number of belief levels are preferably greater than two. The perceptually significant features include color, texture and/or shape. In the preferred embodiment, the main subject is indicated by a continuously valued belief map. The belief values of the main subject are determined by segmenting the image into regions of homogenous color and texture, computing at least one structure feature and at least one semantic feature for each region, and computing a belief value for all the pixels in the region using a Bayes net to combine the features. In an illustrative application, the inventive method is implemented in an image retrieval system. In this implementation, the inventive method automatically stores perceptually significant features of the main subject or background of a plurality of first images in a database to facilitate retrieval of a target image in response to an input or query image. Features corresponding to each of the plurality of stored images are automatically sequentially compared to similar features of the query image. Consequently, the present invention provides an automatic system and method for controlling the feature extraction, representation, and feature-based similarity retrieval strategies of a content-based image archival and retrieval system based on an analysis of main subject and background derived from a continuously valued main subject belief map. <IMAGE>
System and method for determining image similarity