A common method of making a three-dimensional (herein also denoted as “3D”) mathematical model of the surfaces of a scene is to project light reference patterns onto the scene, take an image of the scene showing the projected patterns (herein referred to as “structured illumination”), and then analyze the imaged patterns in comparison with an image of the same patterns projected onto a plane surface. By precisely measuring the respective displacements of the projected patterns in the image of the scene against the positions of the matching patterns in the reference image, it is possible to employ triangulation to determine the positions of the scene surfaces along the axis normal to the image plane, and thus obtain relative 3D positions of the scene surfaces for use in constructing a mathematical model of the scene surfaces. Both regular and random patterns are used in structured illumination.
It is noted that structured illumination is only one of several means of obtaining 3D information about intra-oral scenes. Other means include, but are not limited to: stereo optical imaging; and ultrasound. Various techniques are discussed and disclosed in U.S. Pat. No. 6,402,707 to one of the present inventors. Thus, the background and principles of the present invention are not limited to use in conjunction with structured illumination, but apply to other means of obtaining 3D information about intra-oral scenes, in general herein denoted as “imaging of an intra-oral scene”. It is therefore understood that in the present application, structured illumination is used only as a non-limiting example to illustrate the application and use of embodiments of the present invention. Structured illumination is important in dental applications because of the ability to provide 3D models of intra-oral scenes with a minimal amount of equipment. The interior of the human mouth presents a relatively small cavity in which to insert imaging equipment, and the use of structured illumination, as described above, is well-suited to intra-oral use.
The term “intra-oral scene” herein denotes any collection of intra-oral objects, artifacts, surfaces, or features, which can be visualized and modeled, including, without limitation, both natural and artificial features. Three-dimensional models of intra-oral scenes support various aspects of dental practice.
It is also noted that the 2D image of a scene on which structured illumination patterns have been projected, and from which 3D position data can be derived, is in some sense equivalent to the 3D model which results therefrom through triangulation against the corresponding image of the patterns projected onto a plane surface, because the 3D model mathematically contains substantially the same 3D information as the 2D image correlated with the structured illumination. A similar correspondence exists for other 2D imaging techniques that result in 3D models (such as 2D stereo pairs). In this regard, 2D images containing 3D information, such as 2D images of the structured illumination, and the corresponding mathematical 3D models are, to some extent, interchangeable. Therefore, in the present application, references to a “2D image” of a scene and references to a “3D model” of the scene derived from the 2D image are understood to refer to similar data, the only difference being that the data points inherent in the 2D image of the scene are not processed, whereas the data points inherent in the corresponding 3D model have been processed. The term “2D imaging system” herein denotes the apparatus for capturing a 2D image of a scene, including, but not limited to those illuminated by patterns of structured illumination; and the term “3D imaging system” herein denotes a 2D imaging system in conjunction with apparatus for obtaining 3D information, including, but not limited to apparatus for projecting structured illumination patterns. The processing of 2D position data to obtain 3D positioning information, such as processing the data of structured illumination patterns, is well-known in the prior art and is considered herein as an inherent capability in a 3D imaging system.
A restriction in using the above-described structured illumination method, however, results from the fact that the projected light patterns have a two-dimensional (herein also denoted as “2D”) extent over the scene surfaces. This is necessary in order to provide the capability of matching the light patterns with their corresponding patterns in the reference scene.
As a consequence, there are limitations in the 3D models of intra-oral scenes produced by such a method. In particular, the following limitations are noted:
Edge Imaging Limitations
FIG. 1 illustrates a view of a tooth 101 (conceptualized as a labial surface of a lower incisor) on which is projected a structured illumination pattern extending over an area 103. Area 103 exhibits a perimeter 103a and a center position 103b. Typically, in a normal projection onto a flat surface, area 103 has a square aspect ratio with a side length 103c denoted herein as “S”. As described above, in order to achieve pattern matching, the structured illumination pattern requires a minimum 2D area, here shown as area 103. Center position 103b defines the x-y position of the pattern for modeling work. As a result, the effective portion of tooth 101 which can be modeled by using structured illumination in this fashion is denoted by a boundary 105, which is nominally a distance S/2 from the edge of tooth 101. This limitation is shown in FIG. 1 near an edge 113 of the tooth's occlusal surface. When projecting the structured illumination with the center 103b on boundary 105, such as in a location 107, the position can be accurately determined. When going outside boundary 105, however, such as in a location 109, a portion 111 of the structured illumination is lost, and thus the pattern may not be matchable with the corresponding pattern projected on the reference plane.
The distance s/2 from the edge is not a precisely-determined limit, because there may be cases where the pattern of the structured illumination can be matched over a smaller area. Conversely, if the structured illumination is not projected normally to the surface, the structured illumination will be distorted by a factor of 1/(sin α), where α is the incident angle of the projection. Thus, in cases where the incident angle is less than 90°, the boundary limit will be more than s/2 from the edge of the tooth.
Stitching Discontinuity Limitations
3D objects scenes generally cannot be visualized completely from a single direction, but must be visualized from a number of positions. That is, a 3D model typically must be constructed from the results of a set of separate imaging operations performed in different positions and/or in different directions, each of which results in an incomplete portion of a 3D model. Combining the separate 3D model portions (also referred to as “tiles”) resulting from these separate operations is generally referred to as “stitching”. In order to stitch together the separate 3D model tiles—that is, to stitch the incomplete 3D model tiles derived therefrom—and thereby obtain a unified 3D model, the different separate 3D models need to have a certain amount of overlapping or common features, by which the stitching regions can be identified. If the separate 3D model tiles do not have such common features, there is typically no way to stitch them together to obtain a unified 3D model.
FIG. 2 illustrates the stitching limitations resulting from using structured illumination for intra-oral 3D imaging. A tooth 201 (conceptualized as a lower incisor) is shown in side view, with a labial surface 203 and a lingual surface 205. A projection system (not shown) projects structured illumination patterns (not shown) onto the surfaces for imaging with imaging systems capable of capturing images of the structured illumination for comparison with the patterns against an image of those patterns on a reference surface device, as previously discussed.
An imaging system in a position 215 is capable of capturing 3D information of surface 203 in a region 207. An imaging system in a position 219 is capable of capturing 3D information of surface 205 in a region 209. However, because of the edge imaging and surface area limitations as discussed above, a region 211 and a region 213 adjacent to the edge of the tooth's occlusal surface cannot be satisfactorily imaged. Likewise, an imaging system in a position 217 is generally incapable of rendering any images in a region 221 connecting labial surface 203 with lingual surface 205. In such a case, there is no way to stitch the images produced by a 3D imaging system in position 215 with the images produced by a 3D imaging system in position 219. Thus, the use of structured illumination for intra-oral 3D imaging may result in discontinuities that make stitching difficult or impossible.
It is noted that the same imaging system may be used in different positions. Thus, in FIG. 2, the same imaging system may be sequentially used in positions 215, 217, and 219. It is the positions of the imaging system which are necessarily different for multiple 3D model imaging, rather than the imaging systems themselves. Similarly, the other figures of the present application relate to different imaging system positions, rather than different imaging systems themselves. Of course, separate imaging systems may also be employed for each different position without limitation in this regard.
Limitations in Imaging and Modeling of Special Features
Small objects and other special intra-oral features often lack sufficient surface area for satisfactory 3D reconstruction from projection of structured illumination patterns. Even if 3D reconstruction is achieved, there may be insufficient 3D area in the tile to enable stitching to another tile. In addition, if a feature has a polished or reflective surface, structured illumination patterns typically are not visible when projected onto such features.
This limitation is illustrated in FIG. 3, for an abutment 307, shown as being located in position for the placement of a crown to replace a missing lower incisor. Abutment 307 is affixed to an implant (not shown), and examples of abutment 307 include, but are not limited to: implant abutments; healing abutments; healing caps; and impression abutments. Teeth 301, 303, and 305 can be imaged using structured illumination, and their positions thus determined Abutment 307, however is considerably smaller than the teeth, and is comparable in size to area 103 for the projected patterns. In addition, abutment 307 may be reflective or otherwise not suitable for pattern projection. In this manner, structured illumination cannot be used to determine the precise position of abutment 307, and this is another limitation of the methodology.
Some attempts to overcome the imaging limitations of the prior art were described in U.S. Pat. No. 6,925,198, to Scharlack et al. ‘198’ describes a method and system for creating three-dimensional models of implant-bearing dental arches, and other anatomical fields of view, employs three-dimensional scanning means to capture images of an anatomical field of view wherein there have been positioned (and preferably affixed to an anatomical feature) one or more three-dimensional recognition objects having a known geometry, such as a pyramid or a linked grouping of spheres. Image processing software is employed to locate and orient said recognition objects as reference data for stitching multiple images and thereby reconstructing the scanned field of view. Recognition objects placed in areas of low feature definition enhance the accuracy of three-dimensional modeling of such areas.
US Patent Application Publication No. 2008/0002869 to Scharlack et al., describes a three-dimensional-based modeling method and system designed for dentistry and related medical (and appropriate non-medical) applications. Data capture means produces a point cloud representing the three-dimensional surface of an object (e.g., dental arch). Three-dimensional recognition objects are provided, particularly within those areas in the image field that have low image definition, and particularly in such of these areas that appear in overlapping portions of at least two images, to provide the three-dimensional image processing software with position, angulation, and orientation information sufficient to enable highly accurate combining (or “stitching”) of adjoining and overlapping images. Alignment, and creation of aligned related objects or models thereof, such as maxillar and mandibular arches, is facilitated.
There is thus a need for, and it would be highly-desirable to have, apparatus and methods for use in 3D modeling of intra-oral features in dental applications that overcome the aforementioned limitations of structured illumination and other means of 2D imaging. This goal is met by the present invention.