Hands-On Deformation of Volumetric Anatomical Images on a Touchscreen

In this work, we propose a novel metaphor to interact with volumetric anatomical images, e.g., magnetic resonance imaging or computed tomography scans. Beyond simple visual inspection, we empower users to reach the visible anatomical elements directly with their hands, and then move and deform them through natural gestures, while respecting the mechanical behavior of the underlying anatomy. This interaction metaphor relies on novel technical methods that address three major challenges: selection of anatomical elements in volumetric images, mapping of 2D manipulation gestures to 3D transformations, and real-time deformation of the volumetric images. All components of the interaction metaphor have been designed to capture the user's intent in an intuitive manner, solving the mapping from the 2D touchscreen to the visible elements of the 3D volume. As a result, users have the ability to interact with medical volume images much like they would do with physical anatomy, directly with their hands.

Automated summary

This work proposes a novel metaphor for interacting with volumetric anatomical images, enabling users to manipulate visible anatomical elements directly with their hands through natural gestures while respecting the mechanical behavior of the underlying anatomy. The interaction metaphor addresses challenges such as selecting anatomical elements in volumetric images, mapping 2D manipulation gestures to 3D transformations, and real-time deformation of the volumetric images. The components are designed to intuitively capture user intent and solve the mapping from 2D touchscreen to visible elements of the 3D volume.