Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles

This paper proposes a vision-based action recognition framework that considers the sequential working patterns of earthmoving excavators for automated cycle time and productivity analysis. The sequential patterns of visual features and operation cycles are incorporated into the action recognition framework, which includes three main processes: excavator detection, excavator tracking, and excavator action recognition. The first process uses a pre-trained detector to localize earthmoving excavators on sites from all of captured images. Next, the detection results are associated and the excavators are tracked by Tracking-Learning-Detection algorithm. Lastly, to recognize the operation types of the excavators, their sequential patterns of visual features and operation cycles are modeled and trained with a hybrid deep learning algorithm, i.e., Convolutional Neural Networks and Double-layer Long Short Term Memory. Three experiments were performed to validate the proposed framework and confirm the positive effects of sequential modeling: (1) an experiment without sequential pattern analysis, (2) an experiment with the sequential pattern analysis of visual features, and (3) an experiment with the sequential pattern analysis of visual features and operation cycles. In the experiments, the research team used a total of 72,365 images collected from actual earthmoving sites. The average accuracies of the excavator action recognition in the three cases were 79.8%, 90.9%, and 93.8% respectively. The results demonstrated the applicability of the proposed framework and the significant positive impacts of sequential pattern modeling on the recognition performance. This research contributed to the identification of critical elements that explain sequential working patterns and to the development of a novel vision-based action recognition framework. In addition, the findings of this study can help to automate cycle time analysis and productivity monitoring of earthmoving excavators.