one fundamental task in cranio - maxillofacial ( cmf ) surgery is to restore normal skeletal anatomy in patients with extensive fractures of the facial skeleton and mandible from gunshot wounds , work - related injuries , natural disasters , or traffic accidents .
any attempt to restore a bone fragment to its original position poses considerable risk for additional damage to vital anatomical structures .
furthermore , small errors in the positioning of each fragment may accumulate and result in inadequate reconstruction which in turn may result in poor function and a poor esthetic result . surgical planning from ct data
but if the planning relies only on visual cues , object contact and object penetration can be difficult to discern because contact surfaces are likely to be occluded by the many bone fragments .
current commercially available cmf surgery planning systems , for example systems by planmeca , materialise , and brainlab , rely primarily on two - dimensional graphical interfaces .
these put great demands on the user , requiring that he or she be able to visualize complex 3d models from their 2d projections on a two - dimensional display and be able to plan delicate 3d procedures using a set of 2d projections and 2d interaction tools . furthermore , while a surgeon relies heavily on his / her sense of touch in the operation room , surgical planning systems generally do not use the sense of touch to complement the visual interface . as a result ,
cmf surgery planning is time - consuming and cumbersome , making it difficult to find an optimal surgical strategy , which in turn may result in a less than perfect reconstruction with needless patient discomfort and loss of functionality .
describe interactive computer - assisted planning and surgery tools using photorealistic imaging for optimized treatment of oral and maxillofacial malignancies , and for tissue engineering of bone .
[ 6 , 7 ] describe planning tools that include skull and soft tissue segmentation , assessment of skeletal muscle properties , characterization of the mechanical response of soft facial tissue , clinical validation , and transfer of the cmf planning into the operating room .
haptics has the potential to improve surgical planning by giving the surgeon virtual tools that are familiar from the operating room : s / he can feel if two bone fragments fit together or if the occlusion ( bite ) is correct .
contact forces also help the surgeon to avoid interpenetration of fragments that may be difficult to discern visually .
present a requirements study for cmf surgery planning with haptic interaction for bone fragment and plate alignment , exploring what features might be important in haptic cranio - maxillofacial planning .
this is done with physical mock - ups , complemented by the implementation of some features in software .
haptic fidelity  as a highly important aspect for success in this type of system .
haptic feedback is used to increase the realism in simulators for training of specific surgical procedures .
pettersson et al .   present a simulator for cervical hip fracture surgery training which provides visuo - haptic feedback of the drilling task central to this procedure .
describe a bone surgery training simulator also with focus on drilling , in this case of the temporal bone and the mandible .
a survey of visuo - haptic systems for surgical training with a focus on laparoscopic surgery can be found in .
we present a system that combines stereoscopic 3d visualization with six - dof haptic rendering that can be used by a surgeon with only minimal training .
look - around  in the graphical scene , a simulated spring coupling between the manipulated virtual bone fragment and the haptic handle for enhanced haptic stability , high - precision collision detection , the ability to group and manipulate a set of fragments as one entity , and snap - to - fit , a tool for precision alignment of matching bone fragments .
the patient data comprise segmented volumetric ct data from the fractured regions in which independent bone fragments are labeled .
 ) a half - transparent mirror with stereo glasses gives the user a stereoscopic view of the data , and the haptic unit , positioned under the mirror , has a handle for moving the entire ct model or individual bone fragments .
a head tracker , which continually updates the user s vantage point , gives the user  look - around ,  that is the ability to view objects from different angles by simply moving his / her head .
this is essential for detecting bone fragments that may be ( partially ) occluded from certain vantage points .
a haptic cursor follows the six - dof motions of the haptic handle in the visual space .
when the haptic cursor is in close proximity to a bone fragment ( or group of fragments ) , the fragment or group is graphically highlighted indicating that the user may pick it up and manipulate it ( affect its position and rotation ) by moving the handle with the handle button depressed .
when the haptic cursor is further away from the bone fragments , and no fragment is highlighted , the user may translate and rotate the entire 3d volume , again by moving the handle with its button depressed . 

1the planning system hardware as seen from above ( left ) and from the side ( right ) .
the graphical objects are displayed by the monitor ( a ) and reflected on the half - transparent mirror ( b ) .
the user manipulates the 3d graphical objects with the phantom premium device ( c ) using its handle ( d ) under the half - transparent mirror .
the pushbuttons on the 3dconnexion controller ( e ) under the mirror activate the grouping tool and snap - to - fit fragment surface marking .
the two infra - red cameras ( f ) mounted under the display track the marker rig mounted on the shutter glasses ( g ) for user  look - around . 
the foot - switch ( h ) activates the snap - to - fit attraction forcesfig .
when the haptic cursor is held close to a bone fragment , it is highlighted ( middle ) and the user can then grasp and manipulate it with the six - dof haptic handle ( right ) .
contact forces guide the user during manipulation the planning system hardware as seen from above ( left ) and from the side ( right ) .
the graphical objects are displayed by the monitor ( a ) and reflected on the half - transparent mirror ( b ) .
the user manipulates the 3d graphical objects with the phantom premium device ( c ) using its handle ( d ) under the half - transparent mirror .
the pushbuttons on the 3dconnexion controller ( e ) under the mirror activate the grouping tool and snap - to - fit fragment surface marking .
the two infra - red cameras ( f ) mounted under the display track the marker rig mounted on the shutter glasses ( g ) for user  look - around . 
the foot - switch ( h ) activates the snap - to - fit attraction forces virtual reconstruction of a mandible . each individual bone fragment
when the haptic cursor is held close to a bone fragment , it is highlighted ( middle ) and the user can then grasp and manipulate it with the six - dof haptic handle ( right ) .
contact forces guide the user during manipulation during fragment manipulation , contact force and torque from contacts with other fragments are rendered haptically with high spatial resolution , giving the user an impression similar to that of manipulating a real , physical object around other objects . to limit inter - object penetrations , we simulate a translational and a rotational spring , commonly known as virtual coupling , between the bone fragment currently under manipulation and the haptic handle .
the user may push a manipulated bone fragment toward another bone fragment which stretches the simulated spring , but the manipulated fragment stops at the other fragment s surface instead of penetrating it .
when two or more fragments have been positioned relative to one another , the user may group them and manipulate them as one unit .
additional fragments may subsequently be attached to extend the group and they may also be detached from the group . when bone fragments are grouped , the entire group is given one color .
the grouping tool is activated with pushbuttons on the 3dconnexion unit placed to the left under the half - transparent mirror .
( see fig . 1 . ) in what follows , we describe in more detail the unique feature snap - to - fit , which complements the contact forces to aid the user in bone fragment alignment .
precise alignment of the bone fragments is important since even small rotational and translational errors between fragments may accumulate to much larger errors as a result of the reconstruction of a series of fragments , for example , a mandible with multiple fractures .
but due to visual occlusion , it may be difficult to visually discern the ideal fit between two bone fragments .
just as we use our human haptic ability in the real world to assemble a broken object , six - dof contact forces may provide haptic guidance in finding an optimal fit between two fragments .
however , limited force fidelity in most commercial haptic devices of today makes it difficult to feel when the optimal fit is found as clearly as can be done with real , physical objects .
3alignment of two mandibular fragments using contact forces alone , without snap - to - fit .
visually , the fit may look acceptable from one viewpoint ( left ) , but when seen from the opposite side ( right ) , it is clear that the alignment is sub - optimal .
haptic contact forces provide some guidance , but depending on the quality of the haptic display it may still be difficult to feel the optimal fit alignment of two mandibular fragments using contact forces alone , without snap - to - fit .
visually , the fit may look acceptable from one viewpoint ( left ) , but when seen from the opposite side ( right ) , it is clear that the alignment is sub - optimal .
haptic contact forces provide some guidance , but depending on the quality of the haptic display it may still be difficult to feel the optimal fit the alignment tool , snap - to - fit , complements haptic contact forces in search for a good fit between two bone fragments . for a detailed description of snap - to - fit , we refer the reader to . in summary ,
the user begins by moving a bone fragment close to a matching fracture surface on another bone fragment . from this approximate initial position of the two fragments ,
the user activates snap - to - fit with the foot - switch ( shown in fig .
1 ) that engages attraction forces computed from the fracture surfaces . the forces pull the manipulated objects toward the closest stable fit , that is , it
we scale the attraction forces by the similarity of the fracture surfaces , computed by the colinearity of the surface normals
. fragments with matching surfaces have stronger attraction forces than those with less similar surfaces .
snap - to - fit works best when the fracture surfaces of both fragments are well preserved by the segmentation and the fragments are not too thin or too small .
for some types of fractures , for example , compression fractures , the fracture surface may be damaged with portions of the surface missing . in these cases , a good match between the fracture surfaces may not be possible and the user has to override the system and use his / her expertise to manually find a suitable placement of the bone fragments.fig .
4alignment of two fragments of a mandible using snap - to - fit with fracture surface marking .
the user marks the fracture surfaces using the haptic cursor as a paintbrush ( c , d ) .
additional fragments are aligned to continue the reconstruction of the mandible ( h ) alignment of two fragments of a mandible using snap - to - fit with fracture surface marking .
the user marks the fracture surfaces using the haptic cursor as a paintbrush ( c , d ) .
additional fragments are aligned to continue the reconstruction of the mandible ( h ) one limitation in the original implementation of snap - to - fit is that the fragments may  snap  to several alternative positions , since the whole fragment surface is a potential matching surface .
we therefore extend the method described in   with the ability to mark portions of fragment surfaces as fracture surfaces , allowing the user to  paint  the fracture surfaces with the haptic cursor ( see fig .
4 ) . only painted surfaces are included in the attraction force model , which prevents the fragment from snapping to false regions outside the fragment surface area .
our planning system executes on an hp z400 workstation with an nvidia quadro 4000 graphics processing unit ( gpu ) driving a samsung 120  hz stereo monitor which displays time - multiplexed stereo graphics at a resolution of \documentclass[12pt]{minimal } 
				 \usepackage{amsmath } 
				 \usepackage{wasysym } 
				 \usepackage{amsfonts } 
				 \usepackage{amssymb } 
				 \usepackage{amsbsy } 
				 \usepackage{mathrsfs } 
				 \usepackage{upgreek } 
				 \setlength{\oddsidemargin}{-69pt } 
				 \begin{document}$$1,680\times 1,050$$\end{document } synchronized with a pair of nvidia 3d vision pro   shutter glasses .
the half - transparent mirror rig used for visuo - haptic collocation is manufactured by sensegraphics .
the head tracker is based on an ir optical tracker ( optitrack from natural point ) , with built - in motion capture and image processing , that optically tracks a marker rig consisting of four ir - reflecting spherical markers placed asymmetrically on the stereo glasses worn by the user .
after careful registration of the tracking frame with the visual frame , the head tracker estimates the user vantage point from which we render the stereo perspective .
we render the bone fragment surfaces on the visual display using splatting   which is implemented on the gpu to achieve real - time rendering .
we use a sensable phantom premium 1.5 high force/6dof haptic device , with six - dof ( in / out ) running at a haptic frame rate of 1  khz .
we render the six - dof contact forces using a rigid body contact model combined with a virtual spring ( static virtual coupling ) which decouples the manipulated bone fragment position and orientation from the haptic handle to improve the haptic stability .
we rely heavily on pre - computation and hierarchical data - structures in order to achieve real - time haptic interaction rates . the contact force model and the static virtual coupling
snap - to - fit is implemented according to   with the extension that the user may mark fracture surface areas . only marked surface areas
we load the patient - specific volumetric image data from a dicom stack of ct images .
the images in this study have a resolution of 0.35  mm , and the inter - slice distance is 0.60  mm .
we first segment bone tissue from soft tissue by thresholding the ct data and then remove small , isolated tissue components with fewer than 100 connected voxels using the bwareaopen filter in matlab .
we manually segment and label individual bone fragments from the resulting image volume using itk - snap   before loading them into the planning system .
the patient data comprise segmented volumetric ct data from the fractured regions in which independent bone fragments are labeled .
 ) a half - transparent mirror with stereo glasses gives the user a stereoscopic view of the data , and the haptic unit , positioned under the mirror , has a handle for moving the entire ct model or individual bone fragments .
a head tracker , which continually updates the user s vantage point , gives the user  look - around ,  that is the ability to view objects from different angles by simply moving his / her head .
this is essential for detecting bone fragments that may be ( partially ) occluded from certain vantage points .
a haptic cursor follows the six - dof motions of the haptic handle in the visual space .
when the haptic cursor is in close proximity to a bone fragment ( or group of fragments ) , the fragment or group is graphically highlighted indicating that the user may pick it up and manipulate it ( affect its position and rotation ) by moving the handle with the handle button depressed .
when the haptic cursor is further away from the bone fragments , and no fragment is highlighted , the user may translate and rotate the entire 3d volume , again by moving the handle with its button depressed . 

1the planning system hardware as seen from above ( left ) and from the side ( right ) .
the graphical objects are displayed by the monitor ( a ) and reflected on the half - transparent mirror ( b ) .
the user manipulates the 3d graphical objects with the phantom premium device ( c ) using its handle ( d ) under the half - transparent mirror .
the pushbuttons on the 3dconnexion controller ( e ) under the mirror activate the grouping tool and snap - to - fit fragment surface marking .
the two infra - red cameras ( f ) mounted under the display track the marker rig mounted on the shutter glasses ( g ) for user  look - around . 
the foot - switch ( h ) activates the snap - to - fit attraction forcesfig .
when the haptic cursor is held close to a bone fragment , it is highlighted ( middle ) and the user can then grasp and manipulate it with the six - dof haptic handle ( right ) .
contact forces guide the user during manipulation the planning system hardware as seen from above ( left ) and from the side ( right ) .
the graphical objects are displayed by the monitor ( a ) and reflected on the half - transparent mirror ( b ) .
the user manipulates the 3d graphical objects with the phantom premium device ( c ) using its handle ( d ) under the half - transparent mirror .
the pushbuttons on the 3dconnexion controller ( e ) under the mirror activate the grouping tool and snap - to - fit fragment surface marking .
the two infra - red cameras ( f ) mounted under the display track the marker rig mounted on the shutter glasses ( g ) for user  look - around . 
the foot - switch ( h ) activates the snap - to - fit attraction forces virtual reconstruction of a mandible . each individual bone fragment
when the haptic cursor is held close to a bone fragment , it is highlighted ( middle ) and the user can then grasp and manipulate it with the six - dof haptic handle ( right ) .
contact forces guide the user during manipulation during fragment manipulation , contact force and torque from contacts with other fragments are rendered haptically with high spatial resolution , giving the user an impression similar to that of manipulating a real , physical object around other objects . to limit inter - object penetrations , we simulate a translational and a rotational spring , commonly known as virtual coupling , between the bone fragment currently under manipulation and the haptic handle .
the user may push a manipulated bone fragment toward another bone fragment which stretches the simulated spring , but the manipulated fragment stops at the other fragment s surface instead of penetrating it .
when two or more fragments have been positioned relative to one another , the user may group them and manipulate them as one unit .
additional fragments may subsequently be attached to extend the group and they may also be detached from the group . when bone fragments are grouped , the entire group is given one color .
the grouping tool is activated with pushbuttons on the 3dconnexion unit placed to the left under the half - transparent mirror .
( see fig . 1 . ) in what follows , we describe in more detail the unique feature snap - to - fit , which complements the contact forces to aid the user in bone fragment alignment .
precise alignment of the bone fragments is important since even small rotational and translational errors between fragments may accumulate to much larger errors as a result of the reconstruction of a series of fragments , for example , a mandible with multiple fractures .
but due to visual occlusion , it may be difficult to visually discern the ideal fit between two bone fragments .
just as we use our human haptic ability in the real world to assemble a broken object , six - dof contact forces may provide haptic guidance in finding an optimal fit between two fragments .
however , limited force fidelity in most commercial haptic devices of today makes it difficult to feel when the optimal fit is found as clearly as can be done with real , physical objects .
3alignment of two mandibular fragments using contact forces alone , without snap - to - fit .
visually , the fit may look acceptable from one viewpoint ( left ) , but when seen from the opposite side ( right ) , it is clear that the alignment is sub - optimal .
haptic contact forces provide some guidance , but depending on the quality of the haptic display it may still be difficult to feel the optimal fit alignment of two mandibular fragments using contact forces alone , without snap - to - fit .
visually , the fit may look acceptable from one viewpoint ( left ) , but when seen from the opposite side ( right ) , it is clear that the alignment is sub - optimal .
haptic contact forces provide some guidance , but depending on the quality of the haptic display it may still be difficult to feel the optimal fit the alignment tool , snap - to - fit , complements haptic contact forces in search for a good fit between two bone fragments . for a detailed description of snap - to - fit , we refer the reader to . in summary , the user begins by moving a bone fragment close to a matching fracture surface on another bone fragment . from this approximate initial position of the two fragments ,
the user activates snap - to - fit with the foot - switch ( shown in fig .
1 ) that engages attraction forces computed from the fracture surfaces . the forces pull the manipulated objects toward the closest stable fit , that is , it
we scale the attraction forces by the similarity of the fracture surfaces , computed by the colinearity of the surface normals
. fragments with matching surfaces have stronger attraction forces than those with less similar surfaces .
snap - to - fit works best when the fracture surfaces of both fragments are well preserved by the segmentation and the fragments are not too thin or too small .
for some types of fractures , for example , compression fractures , the fracture surface may be damaged with portions of the surface missing . in these cases , a good match between the fracture surfaces may not be possible and the user has to override the system and use his / her expertise to manually find a suitable placement of the bone fragments.fig .
4alignment of two fragments of a mandible using snap - to - fit with fracture surface marking .
the user marks the fracture surfaces using the haptic cursor as a paintbrush ( c , d ) .
additional fragments are aligned to continue the reconstruction of the mandible ( h ) alignment of two fragments of a mandible using snap - to - fit with fracture surface marking .
the user marks the fracture surfaces using the haptic cursor as a paintbrush ( c , d ) .
additional fragments are aligned to continue the reconstruction of the mandible ( h ) one limitation in the original implementation of snap - to - fit is that the fragments may  snap  to several alternative positions , since the whole fragment surface is a potential matching surface .
we therefore extend the method described in   with the ability to mark portions of fragment surfaces as fracture surfaces , allowing the user to  paint  the fracture surfaces with the haptic cursor ( see fig .
4 ) . only painted surfaces are included in the attraction force model , which prevents the fragment from snapping to false regions outside the fragment surface area .
our planning system executes on an hp z400 workstation with an nvidia quadro 4000 graphics processing unit ( gpu ) driving a samsung 120  hz stereo monitor which displays time - multiplexed stereo graphics at a resolution of \documentclass[12pt]{minimal } 
				 \usepackage{amsmath } 
				 \usepackage{wasysym } 
				 \usepackage{amsfonts } 
				 \usepackage{amssymb } 
				 \usepackage{amsbsy } 
				 \usepackage{mathrsfs } 
				 \usepackage{upgreek } 
				 \setlength{\oddsidemargin}{-69pt } 
				 \begin{document}$$1,680\times 1,050$$\end{document } synchronized with a pair of nvidia 3d vision pro   shutter glasses . the half - transparent mirror rig used for visuo - haptic collocation is manufactured by sensegraphics .
the head tracker is based on an ir optical tracker ( optitrack from natural point ) , with built - in motion capture and image processing , that optically tracks a marker rig consisting of four ir - reflecting spherical markers placed asymmetrically on the stereo glasses worn by the user .
after careful registration of the tracking frame with the visual frame , the head tracker estimates the user vantage point from which we render the stereo perspective .
we render the bone fragment surfaces on the visual display using splatting   which is implemented on the gpu to achieve real - time rendering .
we use a sensable phantom premium 1.5 high force/6dof haptic device , with six - dof ( in / out ) running at a haptic frame rate of 1  khz .
we render the six - dof contact forces using a rigid body contact model combined with a virtual spring ( static virtual coupling ) which decouples the manipulated bone fragment position and orientation from the haptic handle to improve the haptic stability .
we rely heavily on pre - computation and hierarchical data - structures in order to achieve real - time haptic interaction rates . the contact force model and the static virtual coupling
snap - to - fit is implemented according to   with the extension that the user may mark fracture surface areas . only marked surface areas
we load the patient - specific volumetric image data from a dicom stack of ct images .
the images in this study have a resolution of 0.35  mm , and the inter - slice distance is 0.60  mm .
we first segment bone tissue from soft tissue by thresholding the ct data and then remove small , isolated tissue components with fewer than 100 connected voxels using the bwareaopen filter in matlab .
we manually segment and label individual bone fragments from the resulting image volume using itk - snap   before loading them into the planning system .
we invited an experienced cmf surgeon , who did not have any prior experience with our system , to plan two trauma cases .
he first received 45  min of training , which consisted of planning the reconstruction of a patient with the facial fractures shown in fig . 5 ; we denote this case as the practice case . during the practice planning
, we gave oral instructions of how to use the haptic device , head tracker , all features of the system , and supported the surgeon in the planning process .
when the training was completed , we asked him to complete , on his own , a plan of the complex trauma case denoted the evaluation case . in the evaluation case , we focused on the reconstruction of the mandible and disregarded some additional fractures , such as the zygomatic bone and orbit on the left hand side , which should be addressed in a complete plan.fig .
5practice case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
snap - to - fit was used to position the zygomatic bone ( red fragment ) into its correct place practice case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
snap - to - fit was used to position the zygomatic bone ( red fragment ) into its correct place a 59-year - old male had sustained major trauma to his left maxillofacial region resulting in a major defect of the horizontal and vertical part of the mandible as well as a fracture of the zygomatic bone with moderate displacement of the total zygomatic complex without any fragmentation .
the training session involved using the features of the system , including snap - to - fit , in order to position the zygomatic complex correctly . to position the zygoma , the surgeon marked the full fracture surface on the zygomatic bone and its corresponding fracture surface on the cranium .
he then moved the bone fragment into an approximate initial position and activated snap - to - fit which produced the result shown in fig .
an 18-year - old male fell from a sky lift 16  m and sustained comminute fractures of the skull , midface , and mandible .
the parasymphyseal region of the mandible exhibited a complex fracture with broad diastasis , small interpositioned bone fragments , and overlapping bone fragments .
the bilateral angular fractures included complex fractures of the vertical and horizontal mandible on the left side with interposition of bone fragments .
the bilateral orbital fractures exhibited minor dislocation on the right side , while the floor and lateral wall were engaged on the left side .
6evaluation case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
the occlusion is not optimal due to interference from dislocated teeth ( 44 , 43 ) which also led to the anterior position of the temporomandibular joints in the joint space evaluation case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
the occlusion is not optimal due to interference from dislocated teeth ( 44 , 43 ) which also led to the anterior position of the temporomandibular joints in the joint space the surgeon completed the reconstruction shown in fig .
6 in 22  min after 45  min of training on the practice case .
it was not possible to obtain perfect occlusion due to inference from dislocated teeth ; a tool to remove unwanted parts could be useful in such situations . the surgeon made extensive use of the grouping tool to build groups of fragments once he found a good fit .
he also used the head tracking feature more and more throughout the session to look around objects instead of relying on rotation to get good visibility .
he noted that he could perceive haptically when a bone fragment under manipulation did not fit due to misplacement , or due to inadequate reconstruction of previously positioned fragments .
he also commented that the system is useful for understanding the complexity of the specific case , and that he during the planning process gained insights on preferred order of fragment placement ; assembling the fragments in a certain order may provide valuable clues toward the best global reconstruction .
the surgeon did not favor snap - to - fit in this case after trying it on some fragments that were too small to give a robust result .
the surgeon relied , therefore , on contact forces and visual inspection to complete the reconstruction .
a 59-year - old male had sustained major trauma to his left maxillofacial region resulting in a major defect of the horizontal and vertical part of the mandible as well as a fracture of the zygomatic bone with moderate displacement of the total zygomatic complex without any fragmentation .
the training session involved using the features of the system , including snap - to - fit , in order to position the zygomatic complex correctly . to position the zygoma , the surgeon marked the full fracture surface on the zygomatic bone and its corresponding fracture surface on the cranium .
he then moved the bone fragment into an approximate initial position and activated snap - to - fit which produced the result shown in fig .
an 18-year - old male fell from a sky lift 16  m and sustained comminute fractures of the skull , midface , and mandible .
the parasymphyseal region of the mandible exhibited a complex fracture with broad diastasis , small interpositioned bone fragments , and overlapping bone fragments .
the bilateral angular fractures included complex fractures of the vertical and horizontal mandible on the left side with interposition of bone fragments .
the bilateral orbital fractures exhibited minor dislocation on the right side , while the floor and lateral wall were engaged on the left side .
6evaluation case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
the occlusion is not optimal due to interference from dislocated teeth ( 44 , 43 ) which also led to the anterior position of the temporomandibular joints in the joint space evaluation case pre - planning ( first row ) and after the completed virtual restoration ( second row ) .
the occlusion is not optimal due to interference from dislocated teeth ( 44 , 43 ) which also led to the anterior position of the temporomandibular joints in the joint space
the surgeon completed the reconstruction shown in fig . 6 in 22  min after 45  min of training on the practice case .
it was not possible to obtain perfect occlusion due to inference from dislocated teeth ; a tool to remove unwanted parts could be useful in such situations .
the surgeon made extensive use of the grouping tool to build groups of fragments once he found a good fit .
he also used the head tracking feature more and more throughout the session to look around objects instead of relying on rotation to get good visibility .
he noted that he could perceive haptically when a bone fragment under manipulation did not fit due to misplacement , or due to inadequate reconstruction of previously positioned fragments .
he also commented that the system is useful for understanding the complexity of the specific case , and that he during the planning process gained insights on preferred order of fragment placement ; assembling the fragments in a certain order may provide valuable clues toward the best global reconstruction .
the surgeon did not favor snap - to - fit in this case after trying it on some fragments that were too small to give a robust result .
the surgeon relied , therefore , on contact forces and visual inspection to complete the reconstruction .
similar to the success of haptics in surgery training simulators [ 911 ] , we believe that haptics can greatly improve the efficacy of cmf surgery planning software . in order to produce a complete planning tool , we need to add a number of features . a robust automatic method to find
an initial segmentation complemented by interactive segmentation to remove unwanted objects during a planning session , such as the dislocated teeth in the evaluation case , would be of high value .
future work also includes the virtual design of reconstruction plates for additive production prior to surgery   and to explore ways to transfer the reconstruction plan to the operating room .
there is also a need for a function which allows shaping and fitting of bone grafts or biomaterial to repair defects acquired from trauma .
finally , a thorough evaluation with several surgeons including a comparison with existing cmf planning software packages is needed to establish the efficacy of our system .
we have described work in progress on a system that supports the planning of skeletal anatomy restoration in complex trauma cases .
the key features are as follows : stereo graphics and head tracking that enables  look - around  to allow the user to graphically view the patient - specific anatomy in 3d from different angles by simply moving his / her head ; stable six - dof high - precision haptic rendering that provides intuitive guidance when manipulating virtual bone fragments , allowing the user to feel when one fragment is in contact with another ; and snap - to - fit that complements the contact forces with attraction forces to aid the precise placement of larger fragments such as the zygomatic bone .
preliminary testing with one surgeon indicates that our haptic planning system has the potential to become a powerful tool that with little training allows a surgeon to complete a complex cmf surgery plan in a short amount of time .
 purpose  cranio - maxillofacial ( cmf ) surgery to restore normal skeletal anatomy in patients with serious trauma to the face can be both complex and time - consuming . 
 but it is generally accepted that careful pre - operative planning leads to a better outcome with a higher degree of function and reduced morbidity in addition to reduced time in the operating room . however , today s surgery planning systems are primitive , relying mostly on the user s ability to plan complex tasks with a two - dimensional graphical interface . 
 methods  a system for planning the restoration of skeletal anatomy in facial trauma patients using a virtual model derived from patient - specific ct data . 
 the system combines stereo visualization with six degrees - of - freedom , high - fidelity haptic feedback that enables analysis , planning , and preoperative testing of alternative solutions for restoring bone fragments to their proper positions . 
 the stereo display provides accurate visual spatial perception , and the haptics system provides intuitive haptic feedback when bone fragments are in contact as well as six degrees - of - freedom attraction forces for precise bone fragment alignment . 
 results  a senior surgeon without prior experience of the system received 45  min of system training . 
 following the training session , he completed a virtual reconstruction in 22  min of a complex mandibular fracture with an adequately reduced result . 
 conclusion  preliminary testing with one surgeon indicates that our surgery planning system , which combines stereo visualization with sophisticated haptics , has the potential to become a powerful tool for cmf surgery planning . with little training 
 , it allows a surgeon to complete a complex plan in a short amount of time . 
Introduction and related work
Methods
System overview
Snap-to-fit, a complement to contact forces
Hardware and implementation details
Image data handling
System evaluation
Practice case
Evaluation case
Evaluation results and observations
Discussion
Conclusions
Conflict of Interest