  Accurate segmentation of surgical instrument tip is an important task for
enabling downstream applications in robotic surgery, such as surgical skill
assessment, tool-tissue interaction and deformation modeling, as well as
surgical autonomy. However, this task is very challenging due to the small
sizes of surgical instrument tips, and significant variance of surgical scenes
across different procedures. Although much effort has been made on visual-based
methods, existing segmentation models still suffer from low robustness thus not
usable in practice. Fortunately, kinematics data from the robotic system can
provide reliable prior for instrument location, which is consistent regardless
of different surgery types. To make use of such multi-modal information, we
propose a novel visual-kinematics graph learning framework to accurately
segment the instrument tip given various surgical procedures. Specifically, a
graph learning framework is proposed to encode relational features of
instrument parts from both image and kinematics. Next, a cross-modal
contrastive loss is designed to incorporate robust geometric prior from
kinematics to image for tip segmentation. We have conducted experiments on a
private paired visual-kinematics dataset including multiple procedures, i.e.,
prostatectomy, total mesorectal excision, fundoplication and distal gastrectomy
on cadaver, and distal gastrectomy on porcine. The leave-one-procedure-out
cross validation demonstrated that our proposed multi-modal segmentation method
significantly outperformed current image-based state-of-the-art approaches,
exceeding averagely 11.2% on Dice.

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning;Robotics