StNet: Local and Global Spatial-Temporal Modeling for Action Recognition
Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Li-Min Wang, Shilei Wen
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/hyperfraise/StNetpytorch★ 0
- github.com/2023-MindSpore-1/ms-code-217/tree/main/stnetmindspore★ 0
- github.com/kingcong/stnetmindspore★ 0
- github.com/mindspore-ai/models/tree/master/research/cv/stnetmindspore★ 0
- github.com/BigLazyPig/Pytorch-StNet-Full-Implementpytorch★ 0
- github.com/MindSpore-paper-code-2/code3/tree/main/stnetmindspore★ 0
- github.com/2023-MindSpore-4/Code7/tree/main/stnetmindspore★ 0
- github.com/hyperfraise/Pytorch-StNetpytorch★ 0
Abstract
Despite the success of deep learning for static image understanding, it remains unclear what are the most effective network architectures for the spatial-temporal modeling in videos. In this paper, in contrast to the existing CNN+RNN or pure 3D convolution based approaches, we explore a novel spatial temporal network (StNet) architecture for both local and global spatial-temporal modeling in videos. Particularly, StNet stacks N successive video frames into a super-image which has 3N channels and applies 2D convolution on super-images to capture local spatial-temporal relationship. To model global spatial-temporal relationship, we apply temporal convolution on the local spatial-temporal feature maps. Specifically, a novel temporal Xception block is proposed in StNet. It employs a separate channel-wise and temporal-wise convolution over the feature sequence of video. Extensive experiments on the Kinetics dataset demonstrate that our framework outperforms several state-of-the-art approaches in action recognition and can strike a satisfying trade-off between recognition accuracy and model complexity. We further demonstrate the generalization performance of the leaned video representations on the UCF101 dataset.