A Novel Evaluation Model for a Mixed-Reality Surgical Navigation System: Where Microsoft HoloLens Meets the Operating Room

HoloLens-based mixed-reality surgical navigation system (MR-SNS) technology has made great progress. However, the methodology for evaluating users' perceptions concerning the safety, comfort, and efficiency of MR-SNS is still in its infancy. This study was intended to develop a method to systematically evaluate an existing MR-SNS system during actual clinical applications. This method differs from other existing methods currently used in industry, education, and device maintenance. Based on analytical hierarchy process theory and ergonomics evaluation methods, in this article, we propose a novel multicriteria evaluation model for a HoloLens-based MR-SNS. The model includes factors such as comfort, safety, and effectiveness, and is performed in an actual clinical application. A comprehensive experimental platform and scoring system that can analyze all indicators was built. The validation test showed no statistically significant differences in the accuracy of the 3 different movement patterns (P = .95, P > .05). However, the static pattern showed the best accuracy. In addition, no significant difference (P = .68, P > .05) in accuracy was found under 4 kinds of illuminance. A comparison of the results of this evaluation model and the input from experts who use the HoloLens-based MR-SNS in hospitals, indicated that this model has good precision (100%), recall (80%), and F1-measure (88.89%). The results highlighted the full efficacy of the proposed model in determining whether this system can be used in clinical trials to provide indicators for preliminary ex ante feasibility studies. This article describes the lessons learned from conducting this evaluation study of MR-SNS as part of the design process.