The present invention relates to a stereoscopic image processing apparatus for converting space data described in a language dealing with 3D virtual spaces, e.g., a VRML (Virtual Reality Modeling Language), into image data that can be stereoscopically viewed, a stereoscopic vision parameter setting apparatus, and a stereoscopic image display apparatus, and methods for these apparatuses.
With the recent advances in the performance of computers, 3D computer graphics (CG) have been rapidly popularized. Along with this trend, the WWW (Word Wide Web) on the Internet has rapidly become widespread, and the VRML (virtual Reality Modeling Language) has been standardized as a language dealing with a 3D space on the WWW and widely used.
The latest version of this language, which has been currently released, VRML2.0, is used for the construction of a virtual mall for electronic commerce, management of 3D data such as CAD data in an intranet, and the like.
The present applicant has developed a rear-cross lenticular type 3D display (3D display will be referred to as a stereoscopic display apparatus to distinguish it from a 2D display apparatus) as a stereoscopic display apparatus that allows the observer to directly perceive depth. Many stereoscopic display apparatus, other than this display apparatus proposed by the present applicant, have been proposed.
To display a 3D space described in VRML2.0, a VRML viewer apparatus is required. General VRML viewer apparatuses are just 2D display apparatuses (to be referred to as xe2x80x9ctwo-dimensional display apparatusesxe2x80x9d hereinafter in association with xe2x80x9cstereoscopic display apparatusesxe2x80x9d) which uses general monitors to display final images although the described space is a 3D space.
This is because the conventional VRML apparatus cannot efficiently generate images for stereoscopic vision, and hence is not suited for real-time reproduction of stereoscopic images. In addition, some parameters for implementing stereoscopic vision (to be referred to as stereoscopic vision parameters hereinafter) are not supported by VRML2.0, and hence stereoscopic vision cannot be realized in practice.
Stereoscopic vision parameters are parameters associated with base line length, convergence angle, field of view, and the like. In stereoscopically displaying a target such as a 3D geometric model, virtual space, or virtual object, when the unit or value of the size of the target is clear, the target can be displayed to allow the observer to perceive the size and obtain stereoscopic vision. If, for example, an erroneous base line length is set, stereoscopic vision cannot be ensured. It is therefore important to properly set the initial values of stereoscopic vision parameters.
In consideration of this point, the present applicant has proposed a technique of setting the initial value of base line length based on the whole size of a 3D space so as to ensure stereoscopic vision, and setting the convergence angle to 0xc2x0 (i.e., parallel vision) in Japanese Patent Laid-Open (Kokai) No. 2000-020757 (Jan. 21, 2000).
Problems associated with stereoscopic vision parameters are not limited to the time of initial setting.
For example, while a walkthrough system is provided for the user who is experiencing a virtual space, the viewpoint conditions (viewpoint position and visual axis direction) for observation of a target dynamically change. For this reason, even if the initial values of base line length and convergence angle are properly set, the base line length and convergence angle must be dynamically changed in accordance with changes in viewpoint conditions as walkthrough progresses.
To meet this demand, the present applicant has proposed a technique of allowing the user to dynamically change the above stereoscopic vision parameters through a graphic user interface provided on the display screen of a stereoscopic display in Japanese Patent Laid-Open No. 2000-020757.
Japanese Patent Laid-Open No. 2000-020757 has proposed initial values and real-time values of stereoscopic vision parameters.
In this conventional technique, however, since stereoscopic vision parameters are set through a user interface, it is impossible to dynamically change the stereoscopic parameters following change of the viewpoint conditions (viewpoint position, visual axis direction). Therefore, it is necessary to stop viewpoint moving operation and manually change the stereoscopic parameters. Also, according to the method of setting initial value of base line length based on the whole size of the space, there is a problem where an optimum value is not always obtained for a visible part of the space.
Setting the initial value of convergence angle to 0xc2x0 can also ensure stereoscopic vision. This, however, does not necessarily provide desired stereoscopic vision.
Note that the challenge of setting the initial vales and real-time values of stereoscopic vision parameters is not limited to a system handling VRML data. This challenge is posed in handling every space data as long as it is 3D space data having no stereoscopic vision parameters set or 3D space data in a virtual space in which stereoscopic image data or stereoscopic vision parameters must be dynamically changed.
The present invention has been made in consideration of the above prior art and techniques proposed by the present applicant, and has as its object to provide an image processing apparatus and parameter setting apparatus which automatically set optimal initial values of stereoscopic vision parameters.
It is another object of the present invention to provide an image processing apparatus and parameter setting apparatus which automatically update the optimal real-time values of stereoscopic vision parameters.
It is still another object of the present invention to provide an image processing apparatus and parameter setting apparatus which set optimal values of stereoscopic vision parameters in accordance with the needs of a user or observer.
In order to achieve the above object, according to a preferred aspect of the present invention, there is provided a stereoscopic image processing apparatus for obtaining stereoscopic vision of an arbitrary object by displaying left- and right-eye images of the object, characterized by comprising:
parameter determining means for determining a stereoscopic vision parameter on the basis of image data of one of the left- and right-eye images and displacement information representing a displacement distance between the left- and right-eye images in a 2D rendering space; and
output means for using the determined stereoscopic vision parameter for stereoscopic vision output of the left- and right-eye images.
If, for example, base line length as a stereoscopic vision parameter is set to the displacement distance in a 3D space, stereoscopic vision is ensured. That is, a stereoscopic vision parameter is set without troubling the user or observer.
According to another preferred aspect of the present invention, the parameter determining means comprises:
displacement distance setting means for presetting a displacement distance between left and right image data of a predetermined object point of the object, which appears when the image data are rendered in the 2D rendering space;
displacement computing means for computing a position of a displaced point separated from coordinates of the object point in the 2D rendering space by the displacement distance in a base line length direction of an observer;
coordinate transformation means for computing coordinates of the displaced point in a 3D space; and
base line length determining means for determining a base line length on the basis of a distance between 3D coordinates of the displaced point and the object point.
The necessity to determine stereoscopic vision parameters becomes noticeable when 3D data having no data about stereoscopic vision parameters is to be handled. Since the initial values of stereoscopic vision parameters must be set for such data, according to still another preferred aspect of the present invention, the stereoscopic vision parameter value determined by the parameter determining means is used as an initial value regardless of the movement of the viewpoint position of the observer.
If the viewpoint condition changes, the way 3D data looks may greatly change. For this reason, recalculation of a stereoscopic vision parameter is required. According to still another preferred aspect of the present invention, the parameter determining means is restarted to recalculate a value of a stereoscopic vision parameter at an arbitrary time point.
The above object point (e.g., a point on a virtual object which is nearest to the observer) must be specified in the 2D rendering space. For this reason, according to still another preferred aspect of the present invention, the object point is regarded as a point that satisfies a predetermined condition, and a plurality of search positions in the rendering space are set to have a discrete distribution to quickly search for the object point in the rendering space.
In addition, according to still another preferred aspect of the present invention, the stereoscopic vision parameter includes at least one of a base line length and convergence angle.
An optimal value of a stereoscopic vision parameter is set because the viewpoint position moves. According to still another preferred aspect of the present invention, therefore, the moving means comprises a sensor for detecting a moving amount of an actual viewpoint position of the observer. This arrangement is suited for walkthrough in a mixed reality space.
According to still another preferred aspect of the present invention, the moving means comprises a sensor for detecting a moving amount of a virtual viewpoint position of the observer. This arrangement is suited for walkthrough in a virtual space using a mouse or the like.
When the moving amount of the viewpoint position of the observer exceeds a predetermined value, recalculation of a stereoscopic vision parameter is often required. According to still another preferred aspect of the present invention, therefore, the parameter determining means is restarted to recalculate a corresponding value of a stereoscopic vision parameter.
The timing of recalculation of a stereoscopic vision parameter should not always be automatically set, but may be set in response to a request from the user. According to still another preferred aspect of the present invention, therefore, the parameter determining means is restarted in response to a request from the observer to recalculate a corresponding value of a stereoscopic vision parameter at the viewpoint position of the observer.
It is an object of the present invention to automatically set an optimal value of a stereoscopic vision parameter. However, this value may be set by the user himself/herself. According to still another preferred aspect of the present invention, therefore, the apparatus further comprises user interface means for manually setting a value of a stereoscopic vision parameter.
In the above apparatus, the mode of automatically setting a stereoscopic vision parameter value may conflict with the mode of allowing the user to set a stereoscopic vision parameter value. According to still another preferred aspect of the present invention, therefore, the apparatus further comprises means for arbitrating between the user interface means and the stereoscopic vision parameter setting means. In the embodiment described later, an icon for designating one of the modes is displayed on the display screen.
According to still another preferred aspect of the present invention, when a convergence angle is to be set as a stereoscopic vision parameter, the parameter determining means obtains a spread of an input virtual space in a depth direction, and determines a vergence angle on the basis of a direction of a substantially a midpoint position in the spread in the virtual space in the depth direction which is viewed from left and right viewpoint positions of the observer.
According to still another preferred aspect of the present invention, the predetermined object point is set at a nearest point on the object which is substantially nearest to the viewpoint position of the observer in a depth direction of a 3D space or a virtual sapce.
According to still another preferred aspect of the present invention, the displacement distance is determined in advance on the basis of a pixel size of a display apparatus on which a stereoscopic image is displayed and a distance from a display screen of the display apparatus to the observer. The parameter that matches the maximum value of the displacement distance determined by these factors, e.g., base line length, guarantees three-dimensional vision.
The displacement distance varies depending on the display apparatuses and the distances from the observer to the screens. According to still another preferred aspect of the present invention, therefore, the apparatus comprises a memory for storing predetermined displacement distance values in advance in accordance with combinations of a plurality of types of display apparatuses that can be connected to the stereoscopic image processing apparatus and distances from display screens to the observer, and
means for detecting a distance from the display screen to the observer, and
the parameter setting means reads out a displacement distance value matching a distance from the connected display apparatus to the observer from the memory.
When the visual axes are parallel at the left and right viewpoint positions, the nearest point on an object in a virtual space exhibits a maximum disparity distance. According to still another preferred aspect of the present invention, therefore, the nearest point is determined to be a pixel having a minimum depth value which is selected from image data obtained by perspective transformation of 3D space data from a viewpoint of the observer""s left- or right-eye.
According to still another preferred aspect of the present invention, the displacement computing means comprises:
means for performing perspective transformation of 3D space data from one of the left- and right-eye viewpoints;
nearest point determining means for determining, as the nearest point, a pixel, of the image data subjected to perspective transformation by the means, which has a minimum depth value;
means for computing a position of a displaced point, in the 2D rendering space, which is separated from coordinates of the pixel having the minimum depth value by the displacement distance in the base line length direction of the observer; and
means for obtaining coordinates of the displaced point in a 3D space by inverse perspective transformation of image at a position of the displaced point in the 2D rendering space from another viewpoint.
A general-purpose workstation or personal computer has a real or virtual z buffer (or depth buffer) in a 3D image processing apparatus. Depth value is held in such a buffer in rendering a 3D image. According to still another preferred aspect of the present invention, therefore, the apparatus further comprises a z buffer memory for storing a depth value to allow the use of the depth value stored in the buffer memory. This makes it possible to reduce the system resources and calculation load.
According to still another preferred aspect of the present invention, a distribution pattern of a plurality of search points in the 2D rendering space is set with a density of search points increasing toward a center so as to search for the nearest point in the 2D rendering space. This is because the observer often sees a central portion, and it suffices if stereoscopic vision is ensured in an area where the observer is gazing.
The nearest point varies in accordance with the distribution of recesses/projections of an object in a virtual space. According to still another preferred aspect of the present invention, therefore, a plurality of distribution patterns of a plurality of search points in the 2D rendering space are set to search for the nearest point in the 2D rendering space, and
the apparatus detects a viewpoint position/posture of the observer and selects a distribution pattern that matches the position/posture.
Furthermore, in order to achieve the above objects, according to the present invention, there is provided a stereoscopic vision parameter setting apparatus for determining a stereoscopic vision parameter for stereoscopically displaying an arbitrary object, characterized by comprising
parameter determining means for determining a stereoscopic vision parameter on the basis of 3D space data to be displayed and displacement information representing a displacement distance between the left- and right-eye images in a 2D rendering space,
wherein the determined stereoscopic vision parameter is stored or output outside to be used for stereoscopic vision afterward.
Preferably, this stereoscopic vision parameter setting apparatus further comprises:
a memory for storing a determined value of a stereoscopic vision parameter; and
means for updating the value of the stereoscopic vision parameter and storing the update result in the memory.
The above objects of the present invention can also be achieved by a stereoscopic image display apparatus comprising the stereoscopic image processing apparatus described above.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.