THPoseLite, a Lightweight Neural Network for Detecting Pose in Thermal Images

Nowadays, smart environments (SEs) enable the monitoring of people with physical disabilities by incorporating activity recognition. Thermal cameras are being incorporated as they preserve privacy. Some deep learning (DL) solutions use the pose of the users because it removes external noise. Although there are robust DL solutions in the visible spectrum (VS), they fail in the thermal domain. Thus, we propose thermal human pose lite (THPoseLite), a convolutional neural network (CNN) based on MobileNetV2 that extracts pose from thermal images (TIs). In a novel way, an auto-labeling approach has been developed. It includes a background removal using an optical flow estimator. It also integrates Blazepose [a pose estimator for VS images (VSIs)] to obtain the poses in the preprocessed TIs. Results show that the preprocessing increases the percentage of detected poses by Blazepose from 19.55% to 76.85%. This allows the recording of human pose estimation (HPE) data sets in the VS without requiring VS cameras or manually annotating data sets. Furthermore, THPoseLite has been embedded in an Internet of Things (IoT) device incorporating an edge tensor processing unit (TPU) accelerator, which can process TIs recorded at 9 frames per second (FPS) in real time (12.28 FPS). It requires fewer than 6W of energy to run. It has been achieved using model quantization, decreasing the accuracy in estimating the poses by only 1%. The mean-squared error of MobileNetV2 in test images is 35.48, obtaining accurate poses in 21% of the images that Blazepose is not able to detect any pose.

Automated summary

In this study, THPoseLite, a convolutional neural network (CNN) based on MobileNetV2, is proposed to extract poses from thermal images (TIs) by pre-processing and utilizing Blazepose. The integration of THPoseLite into an IoT device with an edge tensor processing unit (TPU) accelerator allows real-time processing of TIs, achieving accurate pose estimation with low energy consumption.