Ontology-based human behavior indexing with multimodal video data

Observing and analyzing human behavior is a labor-intensive task. Video data is a crucially important resource for ethnographical study, but the accessibility is limited because describing the semantics of video content is difficult. Moreover, the user typically must watch the entire video archive to identify important findings. This study aims to make human behavior computable and utilize it in various domains such as user-centric manufacturing and safety management. To this end, the work proposes an ontology and a semantic video indexing methodology that integrates human annotation and DNN detector-based annotations of video content and converts them into a knowledge graph. This knowledge graph of ontology-based human actions enables us to apply various computational algorithms to human behaviors. The reported proof of concept system retrieves multimodal data, represents every human behavior annotations as an RDF knowledge graph (KG), and exploits the KG to analyze behavior patterns. As a case study, the work was evaluated in the elderly care domain. A formal notation was used to retrieve complex action sequences with specific conditions. The system was validated to retrieve complex behavior patterns and display the location of such behavioral events in the video.

Automated summary

The study aims to make human behavior computable and applicable in various domains by proposing an ontology and semantic video indexing methodology. By conducting a case study in the elderly care domain, the system was validated to retrieve complex behavior patterns and display the location of such behavioral events in the video.