MARS: Motion-Augmented RGB Stream for Action Recognition

Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearancestreamfor RGB frames and a motion stream for opticalflowframes. Although combiningflow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-worldapplicationsrequiringlow latency. In thispaper we introduce two learning approachesto train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoidsflow computation at test time. First,by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity. Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-basedloss and the standardcross-entropy loss for action recognition. We denote the stream trainedusing this combined loss as MotionAugmented RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics comparedto 72.0% and 65.6% with RGB and Flow streams respectively.