The present invention relates to an image processing system which detects a change in a predetermined region of a facial image and decides a facial expression.
Generally, major elements for deciding a good picture of a photographed person include facial expression, color of the face, lighting and shading of the face, and photographic angle of the face. Among these, the most dominant element is the facial expression of the photographed person. However, since the facial expression always changes, it is difficult to catch the optimal photo opportunity. Further, depending on circumstances, a person to be photographed often cannot express the desired expression, due to not only feelings of tension but also other circumstance in which not-smiling person, on a routine bases, cannot control facial muscles for the desired expression.
In conventional analog photography, the above-described problem has been solved by the photographer guiding the person to be photographed, and also by photographic technology via which the photographer captures the best photo opportunity. Further, when a photographed image has something wrong or also when it is impossible to capture the image again, the solution is re-touching, in which a skilled person modifies an original picture by hand painting. However, such methods of manual image modification can at most enhance a face with closed eyes or wrinkles, but it is difficult to actually change the facial expression.
On the contrary, in an image taken by digital means, or an image which is digitized from an analog image via a scanner, it is relatively easy compared to the analog image to perform image modification after photography. However when major image modification of a facial expression is conducted, the facial image may become quite artificial.
Accordingly, it is preferable that the digitally photographed facial expression is initially the best one possible, like that of the analog photography.
Often the favored facial expression is a smiling face. With respect to a facial expression captured by photographic means, if it is possible to determine a smiling face, it is then possible to detect the optimal photo opportunity, that is, after reading continually changing facial expression as digital images, these interpreted as smiling expressions can be selected and stored in a memory means. Yet further, among plural facial images stored in the memory means, only the images of the smiling faces can be extracted with relative ease.
It is well-known that the facial expression is greatly changed owing to the small changes of shape, position and angle, with respect to an eye, an eyebrow and the mouth, which happen in a time as short as 0.3 sec. Widely proposed are technologies based on the above-described well-known art, by applying digital image processing technology, changing or displacing the fundamental predetermined regions of the facial image, such as an eye, an eyebrow and the mouth, the facial expression of a smile, anger, surprise and sorrow can be formed (See for example, Patent Document 1).
However, in order to determine the expression of a facial image, a great number of data must first be processed with respect to the facial image of an individual person, each of course having different facial characteristics. Next, those characteristics must be selected from the data, from which the chosen expression must be determined. However, practical problems, such as excessively long processing time and the high cost of such an apparatus, still exist.
In portrait photography, when a smiling face, as seemed to be the best representative expression, is to be automatically captured, the above-described problems occur. [Patent Document 1: JP-A 11-167626 (Pages 1-2)]
(JP-A means Japanese Patent Application publication.) [Document 1: Muscle Activity Patterns and Geometrical Features in Attractive Facial Expressions (Sugahara et. al), Proceedings of the 5th Annual Conference of JSKE 2003]