1. Field of Invention
This invention relates to a unique structure and methodology that creates a virtual robotic control system with special application to provide a more effective and natural means for a person to operate one or more stand alone or networked optico-mechanical robotic microscopes.
2. Description of Prior Art
Around 1674 Leeuwenhoek first examined microorganisms in a drop of water. This initiated the field of microscopy whereby a plurality of inert and biologic materials and objects with structures too small to be seen with the naked eye (herein below referred to as Microscopic Objects) could be routinely examined through the use of a magnifying device (herein below referred to as a Microscope) by a person or persons (herein referred to as Microscopist). Viewing Microscopic Objects under the Microscope generally requires that they be made thin enough to allow light to be transmit through them and to be fixed onto a carrier made of glass or other transparent material (herein below referred to as a Slide) and this is most often done by cutting one or a plurality of very thin sections from a larger object, adhering one or more sections to a Slide, staining them with a plurality of reagents, and optionally applying a transparent cover slip to enhance the visual appearance of structural components of the Microscopic Object so as to be more easily viewed by a Microscopist.
This viewing is achieved by placing a Slide on a flat surface known as a Microscope Stage positioned above a focusable light source and adjustable iris and beneath a plurality of magnifying lenses (herein each lens referred to as Objective), each objective providing a different magnification, through which light may be transmitted passing through an Objective rotated into the axis of the light thereby magnifying the Microscopic Object and displaying the resulting light image through a set of oculars manufactured from additional magnifying lenses, to the Microscopists' eyes. A single Microscopist controls the Microscope by moving a Slide across the Microscope Stage surface either by pushing and pulling it by hand or by fixing it in place with a clip and using two rotating controls attached to the Microscope one moving it in the x-axis and the other in the y-axis o the plane defined by the Microscope Stage. Simultaneously fine details of the Microscopic Object residing on the Slide are focused by the Microscopist rotating a pair of compound knobs known in the industry as course and fine focus controls located on both sides of the Microscope resulting in a change in the vertical distance between the Microscope Stage and Objective along the light axis. The act of viewing the Microscopic Object while moving the Slide, changing the Objectives, and focusing the Objectives (herein referred to as Examining) is a highly complex and exacting interaction of hand-eye coordination that determines the speed and thoroughness of the Examination.
Furthermore, Microscopes require that, to assure viewing of the exact same structures of Microscopic Objects residing on a Slide, all Microscopists must be present at the same location as the Slide. The need to have Microscopists located with the Slide to view it has lead to significant problems in reliably characterizing Microscopic Objects reducing both the efficacy and reliability of knowledge about them. Often the Slide or Slides to be examined must be sent physically to other Microscopists residing at another location. This leads to a number of problems including long delays in obtaining complete examination by a plurality of Microscopists, the prevention of real time interaction between each Microscopist who examines the Microscopic Objects, the potential for each Microscopist to view a different structure on a Slide, and the potential for loss of one or more Slides. These limitations and others have acted as a serious hindrance towards establishing and applying knowledge about Microscopic Objects. One of a plurality of specific examples; in the healthcare field, this limitation has adversely affected the development of reliable diagnostic criteria in hematological, surgical, and cytological pathology and prevented their accurate and precise application by pathologists and technologists to human tissues examined under a Microscope.
To solve this problem, the field of Telepathology has been developed by a number of companies including but not limited to Apollo, Aperio, Nikon, and Zeiss. Many of the above said companies implement Telepathology Systems by employing a computer system over a networks, the internet and virtual private networks (herein referred to as Network) controlling a plurality of modified Microscopes. Modified Microscopes (herein referred to as Robotic Microscope) are manufactured with a motorized Microscope Stage (herein referred to as Robotic Stage) that allows the Slide to be attached and moved by another device and set of individual Objectives of different magnification assembled to a motorized housing (herein referred to as Robotic Objectives) that allows each to be moved in turn into the axis of the Microscope sub stage light and moved vertically with respect to the Robotic Stage by another motorized device allowing a plurality of Microscopists to move the Slide, change magnification of and maintain focus on a Microscopic Object residing on a Slide while viewing digital images of part or all of the Microscopic Object on a video display device. This is accomplished through the issuance and receipt (herein referred to as Transmission) of a continuous stream of image and control information (herein referred to as Signals) to and from the above Robotic Microscope and one or more software applications (here in referred to as Robot Control Program) residing on a computer the Robot Control Program running on an operating system (herein referred to as Computer Interface) such as Microsoft Windows connected to one or more Robotic Microscopes by a plurality of means including but not limited to insulated conducting materials, and wireless systems (herein referred to as Connected or Connection). Prior art Telepathology Systems thereby meets the need to have a plurality of Microscopists residing at different locations examine a Slide or Slides simultaneously and avoids having to send Slides physically to each Microscopist in turn thereby also assuring that each is examining the exact same Microscopic Object and the exact same structures making up the Microscopic Object and communicating and discussing their observations through a telephonic system with other Microscopists also Examining the Slide.
Alternately, a solution offered by some of the above said companies such as Nikon involves making one or a plurality of digital scans of an entire Slide and storing the resulting image data in one or a plurality of files for storage, retrieval, Transmission, and Examination. This solution dispenses with the need for using the physical Slide and Robotic Microscope and relies solely on the use of a Robotic Control Program operated through a Computer Interface to act as if the Microscopist is viewing a Slide through a Robotic Microscope. Therefore, for the purpose of clarity, the files containing digital scans of an entire Slide are herein also referred to as Robotic Microscope as well with the understanding that the manipulation of the images residing in these files is carried out virtually and does not imply the use of a physical Robotic Microscope to Examine a physical Slide.
As presently designed and manufactured, both of the above said prior art Telepathology Systems force the Microscopist use a Computer Interface that hinders their ability to Examine a Slide with the same facility and efficiency as they would using a Microscope to Examine a Slide directly requiring that a virtual robotic control system be invented to solve the resulting problems.
A review of prior art virtual robotic controls is rich and extensive with tens of thousands of references and patents in a wide range of fields including but not limited to the following major contributions that bare on this invention: Drones (unpiloted planes) can be operated by a person located a great distance from the plane using a combination of virtual robotic control devices and computer interface. However, unlike the present invention, the person does not sit in a virtual plane and have both the option of flying a drone or flying the virtual plane to the same effect. The same is true in the flight simulation field where the pilot in training is surrounded by a realistic cockpit equipped with numerous virtual robotic controls that effect pitch and yaw while simulating altitude and other flight conditions while video display devices simulate the windows of the aircraft. Yet, again, this prior art does not address the concepts put forward by the present invention since the flight simulation cockpit does not control another real plane in real time nor does is it also capable of flying as well. In the field of radioactive materials and highly infectious organisms robotic handling devices have been employed for many years sometimes utilizing video display devices in association with virtual mechanical hand devices. Unlike the present invention, these devices serve a limited purpose of protecting the user from exposure to highly lethal materials and do not enhance the results of their work or allow a similar device residing at another location to be controlled in real time as the present invention does. Another important area is the use of virtual devices in computer gaming; for example Wii games that provide virtual robotic control devices for the user. However, these are used solely to control graphical images on a video display device for the purpose of play or training and do not lead to the control of another real device for the purpose of carrying out a task. Apple's IPod's revolutionary touch pad capabilities allow the user to grasp virtual images on a video display device and so a set of virtual microscope robotic control devices could be recapitulated on screen. Unfortunately, the IPod requires moving or turning something on a screen, which is flat and so requires the user look at the screen because there are no three dimensional tactile signals that allow the user to know where or how much they have actually moved a control. So, for example this is also the major problem with such prior art as Hidehiko Yamaguchi's Display System (U.S. Pat. No. 6,441,807 B1).
The closest that any virtual robotic control system comes to this invention is found in advanced surgery where a surgeon can manipulate a set of medical instruments through a combination of computer interface that magnify living tissues and virtual robotic control devices that carry out surgery or microsurgery. Oleynikov et al.'s Robot for Surgical Applications (U.S. Pat. No. 7,492,116 B2) is just one of many examples. However this prior art does not allow a plurality of persons residing at different locations to simultaneously participate in the surgery. Another example is Yulun Wang's Minimally Invasive Surgical Training using Robotics and Telecollarboration (U.S. Pat. No. 7,413,565 B2) which comes closer but focus' on providing an experienced surgeon with a means of guiding and overriding a resident in training like the controls in a driver's ed type car. Others like Shawn E. McGinley's Surgical Navigation Instrument useful in Marking Anatomical Structures (U.S. Pat. No. 7,458,997 B2) is focused on enhancing the surgeon's ability to align a prosthetic device for insertion into a femur. Tom Nixon's Tool Grip Calibration for Robotic Surgery (U.S. Patent (U.S. Pat. No. 7,386,365 B2) confines itself to very technical matters related to calibrating jaw forces for surgical and telesurgical systems. Robotic devices sold by Intouch Health do allow a limited means of caring for patients and manipulating objects by an operator residing at another location. However, presently, this is achieved through manipulation of a video display device, keyboard and mouse. None of the above prior art references examination of fixed tissues residing on a glass slide at a distance using a set of virtual robotic controls that recapitulate an actual microscope. Nor does each of the above said prior art provide the unique configurability and flexibility combined with replication of nature movements of the Microscopist in Examining a Slide.
More specifically, one key prior art James W. Bacus' Method and Apparatus for Internet, Intranet, and Local Viewing of Virtual Microscope Slides (U.S. Pat. No. 6,396,941 B1) specifically requires the use of a Keyboard and Mouse and does not cover the concept of a set of virtual robotic controls. Leading companies such as Zeiss have concentrated on the technical aspects of providing high resolution slide images. For example Jack A. Zeineh's System and Method for Creating Magnified Images of a Microscope Slide (U.S. Pat. No. 7,456,377 B2) is an attempt to solve the problem of focusing through a high magnification digital image of a whole Slide as one would do with an actual Slide on a Microscope. Maenle et al. Cytologic Imaging Systems and Methods (U.S. Pat. No. 7,445,935 B2) focus' attention on the technical aspects of holding slides for automated imaging not for robotic systems allowing a Microscopist to Examine them directly. Wetzel et al. System and Method for Finding Regions of Interest for Microscopic Digital Montage Imaging (U.S. Pat. No. 7,421,102) are concerned with artificial intelligence analysis of Microscopic Objects residing on Slides and attempting to locate for further Examination what might be the most important areas; again completely by-passing the critical needs of the Microscopist in having an effective system and methodology for controlling that slide. On a technical level, there is much prior art touching on this invention. However, in each case the actual specifications and claims are for areas not central to this patent. For example Jaeger et al. Rotary Circuit Control Devices with Changeable Graphics (U.S. Pat. No. 5,936,613) introduces the concept of changeable calibration marks on a control device but, unlike the one presented in this invention, does not pertain to the specific Robotic Objective magnification set on a specific Robotic Microscope residing at another location.
what is not encompassed by prior art is the use of a set of virtual robotic control devices that not only allow the execution, in real time, of a set of complex tasks on a plurality of robotic devices residing at a plurality of locations but to allow for the replacement of the very device it is designed to control for use locally, to allow for a plurality of Microscopists to engage in the tasks allowed by prior art Telepathology Systems using highly efficient and effective methods of manipulating the Slide while maintaining appropriate magnification and focus, to allow for a plurality of configurations to suite each Microscopist, and to allow for integration into prior art Telepathology Systems, to name a nonexclusive set of unique improvements.
Specifically, prior art Telepathology Systems impose on the Microscopist unnatural movements in the form of numerous Keyboard strokes, Mouse moves, and Mouse clicks to cause a Slide attached to a Robotic Stage to move thereby limiting the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
Furthermore, prior art Telepathology Systems impose on the Microscopist unnatural movements in the form of numerous Keyboard strokes, Mouse moves, and Mouse clicks to change Robotic Objectives thereby limiting the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
Furthermore, prior art Telepathology Systems impose on the Microscopist unnatural movements in the form of numerous Keyboard strokes, Mouse moves, and Mouse clicks to maintain the Microscopic Object residing on the Slide in fine focus as Microscopist causes a Robotic Stage to move a Slide under a Robotic Objective thereby limiting the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
Furthermore, prior art Telepathology Systems prevents the Microscopist from causing a Robotic Stage to move a Slide while simultaneously maintaining the Microscopic Object residing on the Slide in fine focus causing additional limits to the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
Furthermore, prior art Telepathology Systems impose a design limitation that prevent free hand movement of a Slide, a method most favored by most Microscopists causing additional limits to the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
Furthermore, prior art Telepathology Systems impose a methodology for focusing Robotic Objectives on a Slide that prevent continuous rapid fine up and down movements of the Robotic Objectives, a critical need of the Microscopist to maintain a Microscopic Object residing on the Slide in fine focus during movement of the Slide, as well as building a composite impression of the entire depth of the section thereby greatly hindering the speed, flexibility, thoroughness, and scope of Examination that could be achieved through the use of a Microscope.
For the foregoing reasons and many others to be revealed herein below, there is a need for an improved Virtual Robotic Control System (herein referred to as VRCS) that provides a structure and methodology to control the Slide movement, change of Robotic Objectives, and maintenance of fine focus of Microscopic Objects in a manner that very closely recapitulates that provided to a Microscopist using a Microscope yet still allowing all the benefits herein noted above extant in prior art Telepathology Systems.