A technology has been developed that, based on an image captured by a camera attached to a personal computer (PC), a mobile terminal or the like, determines the position and posture of the camera relative to the captured image. In addition, augmented reality (AR) technology has been developed that displays additional information such as computer graphics (CG) using the position and posture of a camera, and achieves support of a user's work, the additional information being superimposed on a captured image displayed on a screen of a PC, a mobile terminal or the like.
FIG. 10 is an illustration depicting an example AR technology. As depicted in FIG. 10, for instance, when a user takes a picture of a marker 11 and an inspection object 12 using a camera built in a mobile terminal 10, object information 13 for the marker 11 is displayed on a display screen 10a of the mobile terminal 10.
The methods for determining the position and posture of a camera include, for instance, conventional technology 1 using a marker system and conventional technology 2 using a markerless system.
In the conventional technology 1 using a marker system, a marker with a geometric pattern preset in an object or object space is captured by a camera and the position and posture of the camera are calculated based on the shape of the marker. In many cases, a planar square with a side of a predetermined length is used for the marker, and the position and posture of the camera are calculated based on a reference point of the marker.
It is general that the center position of a marker is used as the origin of a three-dimensional coordinate system of the camera.
FIG. 11 is an illustration depicting an example definition of a coordinate system and the position and posture of a camera in a marker system. As depicted in FIG. 11, a marker 15 has reference points 15a, 15b, 15c, and 15d at four corners. Also, an origin 16 of the coordinate system is set to match the origin of a three-dimensional coordinate system of a camera 50.
In the conventional technology 2 using a markerless system, the position and posture of the camera are calculated using, for instance, a characteristic point included in a captured image. In the conventional technology 2, variation in shade is high near a point of attention and a characteristic point is detected based on that the position of the point of attention on the image is uniquely determined by the variation in shade. The conventional technology 2 uses a set of three-dimensional coordinates of characteristic points which have been previously generated. Hereinafter, three-dimensional coordinates of a previously generated characteristic point are referred to as a map point, and a set of map points is referred to as an initial map as appropriate. The conventional technology 2 calculates the position and posture of the camera by bringing a characteristic point present in a currently captured image into correspondence with a map point projected on the captured image.
FIG. 12 is an illustration for explaining the conventional technology 2 that determines the position and posture of a camera. In the example illustrated in FIG. 12, map points S1 to S6 are present. A map point Si is represented by Expression (1) in the world coordinate system. It is assumed that characteristic points x1 to x6 are present on a captured image 20. A characteristic point xi is represented by Expression (2) in a camera coordinate system. Let the map points projected on the captured image 20 be projection points x1′ to x6′. A projection point is represented by Expression (3) in the camera coordinate system.Si=(x, y, z)  (1)xi=(u, v)  (2)xi′=(u′, v′)  (3)
For instance, the conventional technology 2 determines the position and posture of the camera by calculating a camera position posture matrix RT so that the square sum E calculated by Expression (4) has a minimum.
                    E        =                              ∑            p                    ⁢                                                                                    x                  p                  ′                                -                                  x                  p                                                                    2                                              (        4        )            
Next, the conventional technology 2 that generates an initial map will be described. FIG. 13 is an illustration for explaining the conventional technology 2 that generates an initial map. For instance, the principle of stereo photography is used in the conventional technology 2. The conventional technology 2 establishes correspondence between the same characteristic points in two captured images with different image capture positions. The conventional technology 2 generates an initial map in which each of a plurality of corresponding points having a correspondence therebetween is set to a map point based on the positional relationship of the corresponding points in the captured images.
In the example illustrated in FIG. 13, Si denotes a map point to be reconstructed, and a characteristic point xai is defined as the intersection point between the line segment connecting an image capture position Ca of an initial camera and the map point Si, and a first captured image 20a. A characteristic point xbi is defined as the intersection point between the line segment connecting an image capture position Cb of a second camera and the map point Si, and a second captured image 20b. Then, the corresponding points are the characteristic point xai and the characteristic point xbi.
It is general that the camera position and the image-capture direction of the first captured image are applied to the origin of a three-dimensional coordinate system of the initial map. FIG. 14 is an illustration depicting an example definition of a coordinate system and position and posture of a camera in a markerless system. As depicted in FIG. 14, the origin of the three-dimensional coordinate system of the initial map is defined, for instance, relative to the position (Tx, Ty, Tz) of a camera 10 and the image capture direction (Rx, Ry, Rz).
Next, the advantages and disadvantages of the above-described marker system and markerless system will be described. An advantage of the marker system is that its resistance to an environmental change and motion of a camera is high and the robustness when the position and posture of a camera are detected is high. Another advantage of the marker system is that the position and posture of a camera are quickly detectable because a throughput for calculating the position and posture is low.
A disadvantage of the marker system is that when part of a marker is undetectable, the position and posture of a camera may not be calculated. Another disadvantage of the marker system is that when a marker is out of the image capture range of a camera, the position and posture of the camera may not be calculated. Another disadvantage of the marker system is that when a marker is captured from the front, the accuracy of calculation of the position and posture of a camera is reduced due to the calculation principle.
An advantage of the markerless system is that when the number of characteristic points is large, the accuracy in calculating the position and posture of a camera is improved. Another advantage of the markerless system is that when characteristic points are distributed over a wide range in a captured image, the accuracy in calculating the position and posture of a camera is improved. Another advantage of the markerless system is that even when part of characteristic points is undetectable, the position and posture of a camera may be calculated. Another advantage of the markerless system is that even when part of characteristic points is out of the image capture range of a camera, the position and posture of a camera may be calculated.
A disadvantage of the markerless system is that performance in detecting a characteristic point and performance in matching with a map point are susceptible to an environmental change and motion of a camera. Another disadvantage of the markerless system is that when a similar pattern is present around characteristic points or characteristic points themselves are similar, an mistake in matching is likely to occur and the accuracy of calculation of the position and posture of a camera is reduced. Another disadvantage of the markerless system is that when the number of characteristic points is small, the accuracy of calculation of the position and posture of a camera is reduced.
In order to compensate for the aforementioned disadvantages by the aforementioned advantages, a conventional technology 3 has been developed that calculates the position and posture of a camera by each of the marker system and the markerless system and that uses either one of the results of the calculation. In the conventional technology 3, for instance, as illustrated in FIG. 15, the coordinate system of the marker system is set to match the coordinate system of the markerless system so that the coordinate systems are easily compared. FIG. 15 is an illustration depicting an example definition of a coordinate system in the conventional technology 3. For instance, the origin 16 of the marker 15 is set to match the origin of a three-dimensional coordinate system for the map points S1 to S6.
In the conventional technology 3, when only one of the marker system and the markerless system has succeeded in calculating the position and posture of the camera, the result of successful calculation is used. However, when both systems provide successful calculation, criterion for selecting which one of results to be used is important.
For instance, in the conventional technology 3, in some cases, a system with higher priority is determined in advance according to application and an error in the position and posture of the camera is individually evaluated and the position and posture calculated by a system with a lower individual error are used. It is to be noted that the aforementioned conventional technologies are disclosed, for instance, in Japanese Laid-open Patent Publication No. 2013-225245, Japanese Laid-open Patent Publication No. 2000-275013, and Japanese Laid-open Patent Publication No. 2014-92984.