Deep Multimodal Habit Tracking System: A User-adaptive Approach for Low-power Embedded Systems

The pace of population ageing is increasing and is currently becoming one of the challenges our society faces. The introduction of Cyber-Physical Systems (CPS) has fostered the development of e-Health solutions that ease the associated economic and social burden. In this work, a CPS-based solution is presented to partially tackle the problem: a Deep Multimodal Habit Tracking system. The aim is to monitor daily life activities to alert in case of life-threatening situations improving their autonomy and supporting healthy lifestyles while living alone at home. Our approach combines video and heart rate cues to accurately identify indoor actions, running the processing locally in embedded edge nodes. Local processing provides inherent protection of data privacy since no image or vital signs are transmitted to the network, and reduces data bandwidth usage. Our solution achieves an accuracy of more than 80% in average, reaching up to a 95% for specific subjects after adapting the system. Adding heart-rate information improves F1-score by 2.4%. Additionally, the precision and recall for critical actions such as falls reaches up to 93.75%. Critical action detection is crucial due to their dramatic consequences, it helps to reduce false alarms, leading to building trust in the system and reducing economic cost. Also, the model is optimized and integrated in a Nvidia Jetson Nano embedded device, reaching real-time performance below 3.75 Watts. Finally, a dataset specifically designed for indoor action recognition using synchronized video and heart rate pulses has been collected.

Automated summary

The pace of population ageing is increasing and becoming a significant challenge. The introduction of Cyber-Physical Systems (CPS) has facilitated the development of e-Health solutions for alleviating the associated burden. This study presents a CPS-based solution, a Deep Multimodal Habit Tracking system, to monitor and improve the autonomy and healthy lifestyles of individuals living alone at home. The system combines video and heart rate cues for accurate indoor action identification, with local processing ensuring data privacy and reducing bandwidth usage. The solution achieves high accuracy and additional heart rate information improves performance, especially in detecting critical actions like falls. The optimized model is embedded in a Nvidia Jetson Nano device, providing real-time performance with low power consumption. Additionally, a dataset specifically for indoor action recognition using synchronized video and heart rate pulses has been collected.