Endoscopes are elongated devices used to visualize the insides of cavities. Originally, endoscopes were equipped with eyepieces for direct observation. Today many endoscopes are equipped with electronic cameras, such as CCD or CMOS image sensors. These sensors are used to capture images from the area viewed with the endoscope. Endoscopic imaging is the process of capturing images from internal structures and transmitting them to an external viewer.
FIG. 1 illustrates specific definitions related to image capture. An imaging device, in this case an endoscope 10, is pointed towards a surface to be viewed 12 in a viewing direction 14. A principal light ray 16 travels from a point 18 on the surface 12, to the endoscope 10, through the endoscope optics (not shown), and finally to an image sensor (not shown). Prior to being refracted inside the endoscope 10, any principal ray leaving a point on the surface 12 travels approximately towards a viewing point 20. The viewing point 20 is the point from which the view can be thought to have been obtained. A plane 22 can be thought to exist between the surface 12 and the endoscope 10, orthogonal to the viewing direction 14. At the intersection of the plane 22 with the principal ray 16, there exists a point 24 which corresponds to the point 18 on the surface 12. The collection of all such points defined by principal rays defines an image. This image is generally equivalent to the actual image received by the image sensor. Therefore an image can be thought of as existing on this plane 22, although the actual imaging plane is located elsewhere. The endoscope has a field of view 26. In this case, the field of view 26 is shown as rectangular, although it could have any shape. Portions of the surface 12 that lie within the field of view 26 make up a visible area 28. The viewing orientation is the rotational orientation of the view about the viewing direction 14. The viewing set is defined herein as the combination of the viewing point, the viewing direction, and the viewing orientation. The view is what is seen from the viewing set. Although defined in the context of endoscopy, this terminology is also applicable to all other viewing situations.
Generally, a viewing situation involves a three-dimensional surface. However, the captured image is only a two-dimensional entity in an image plane that is generally orthogonal to the viewing direction of the endoscope. The image generated at the image plane is typically displayed to the user as would be seen looking along the viewing direction of the endoscope from the endoscopic viewing point. For comparison, stereo-viewing endoscopes capture two slightly offset images which are used to provide the user with a three-dimensional image. However, they still only provide a view from the viewing point and in the viewing direction of the endoscope. Because the endoscope, the user, and the internal structure being examined exist in an actual three-dimensional world, tying the user to the viewing set of the endoscope limits the way information about the internal structure can be conveyed. The user will often desire to change the viewing set. With existing technology, the only option for the user is to move the endoscope. This is not always convenient or even possible. In these and other instances it would be useful for the user to be able to change the viewing set without changing the actual position of the endoscope. An alternative viewing set could provide a better perspective of the physical surface and give the user a better sense of the relative locations of viewed features. For example, it would be advantageous to be able to use a viewing set which is aligned with the user's physical position instead of the physical position of the endoscope. Other viewing sets might also be desired, as determined by the preferences of the user.
Two methods of volumetric image navigation are described in U.S. Pat. No. 6,167,296 to Shahidi and in U.S. Pat. No. 6,442,417 to Shahidi, et al. These methods both utilize a volumetric data set obtained in a preoperative X-ray or MRI scan to construct a three-dimensional anatomical model in a computer. This model is used to generate two-dimensional perspective projection views of the simulated anatomy. These views may then be compared with an actual endoscopic image. However, although these systems provide a three-dimensional model with a variable viewing set, they can only display the endoscopic image from the viewing set of the endoscope.
Because of this limitation, which is common for all existing endoscopic display systems, the true nature of the viewed area is often not conveyed adequately to the user. It would therefore be desirable to display the endoscopic image in a way that more accurately represents the actual three-dimensional surface from which the image was taken and permits the user to achieve a wide variety of different views of this surface.
Accordingly, the primary object of the present invention is to provide a versatile method of displaying endoscopic images that includes three-dimensional surfaces, variable viewing points, directions, and orientations. It is a further object of this invention to have this method applicable to all endoscopes regardless of type, viewing direction, or image sensor format.