SLIMBRAIN: Augmented reality real-time acquisition and processing system for hyperspectral classification mapping with depth information for in-vivo surgical procedures

Over the last two decades, augmented reality (AR) has led to the rapid development of new interfaces in various fields of social and technological application domains. One such domain is medicine, and to a higher extent surgery, where these visualization techniques help to improve the effectiveness of preoperative and intraoperative procedures. Following this trend, this paper presents SLIMBRAIN, a real-time acquisition and processing AR system suitable to classify and display brain tumor tissue from hyperspectral (HS) information. This system captures and processes HS images at 14 frames per second (FPS) during the course of a tumor resection operation to detect and delimit cancer tissue at the same time the neurosurgeon operates. The result is represented in an AR visualization where the classification results are overlapped with the RGB point cloud captured by a LiDAR camera. This representation allows natural navigation of the scene at the same time it is captured and processed, improving the visualization and hence effectiveness of the HS technology to delimit tumors. The whole system has been verified in real brain tumor resection operations.

Automated summary

SLIMBRAIN is a real-time augmented reality system that can classify and display brain tumor tissue. This system captures and processes hyperspectral images at a rate of 14 frames per second during tumor resection surgery to detect and delimit cancer tissue. The results are shown in an augmented reality visualization overlapped with RGB point cloud captured by a LiDAR camera. This visualization allows for natural navigation and improves the visualization and effectiveness of hyperspectral technology for tumor delimitation. The whole system has been verified in real brain tumor resection surgeries.