STM: SpatioTemporal and Motion Encoding for Action Recognition
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, Junjie Yan
Unverified — Be the first to reproduce this paper.
ReproduceAbstract
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| HMDB-51 | STM (ImageNet+Kinetics pretrain) | Average accuracy of 3 splits | 72.2 | — | Unverified |
| Jester (Gesture Recognition) | STM (Resnet-50, 16 frames) | Val | 96.7 | — | Unverified |
| Something-Something V1 | STM (16 frames, ImageNet pretraining) | Top 1 Accuracy | 50.7 | — | Unverified |
| Something-Something V2 | STM (16 frames, ImageNet pretraining) | Top-1 Accuracy | 64.2 | — | Unverified |
| UCF101 | STM (ImageNet+Kinetics pretrain) | 3-fold Accuracy | 96.2 | — | Unverified |