Integral customer pose estimation using body orientation and visibility mask

The analysis of customer pose is one of the most important topics for marketing. Using the customer pose information, the retailers can evaluate the customer interest level to the merchandise. However, the pose estimation is not easy because of the problems of occlusion and left-right similarity. To address these two problems, we propose an Integral Pose Network (IntePoseNet) which incorporates the body orientation and visibility mask. Firstly, benefiting from the simple gaits in retail store, the body orientation can give the global information of pose configuration. For example, if a person is facing to right, the body orientation indicates the occlusion of his or her left body. Similarly, if a person is facing to the camera, his or her right shoulder is probably at the left side of image. The body orientation is fused with local joint connections by a set of novel Orientational Message Passing (OMP) layers in Deep Neural Network. Secondly, the visibility mask models the occlusion state of each joint. It is tightly related to the body orientation because the body orientation is the main reason of self-occlusion. In addition, occluding object (e.g. shopping basket in retail store environment) detection can also give the clues of visibility mask prediction. Furthermore, the system models the temporal consistency by introducing the optical flow and Bi-directional Recurrent Neural Network. Therefore, the global body orientation, local joint connections, customer motion, occluding objects and temporal consistency are integrally considered in our system. At last, we conduct a series of comparison experiments to show the effectiveness of our system.