Spatio-Temporal Self-Attention Weighted VLAD Neural Network for Action Recognition

Shilei CHENG  Mei XIE  Zheng MA  Siqi LI  Song GU  Feng YANG  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E104-D   No.1   pp.220-224
Publication Date: 2021/01/01
Publicized: 2020/10/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2020EDL0002
Type of Manuscript: LETTER
Category: Biocybernetics, Neurocomputing
Keyword: 
human action recognition,  video representation,  VLAD,  self-attention module,  

Full Text: PDF>>
Buy this Article




Summary: 
As characterizing videos simultaneously from spatial and temporal cues have been shown crucial for video processing, with the shortage of temporal information of soft assignment, the vector of locally aggregated descriptor (VLAD) should be considered as a suboptimal framework for learning the spatio-temporal video representation. With the development of attention mechanisms in natural language processing, in this work, we present a novel model with VLAD following spatio-temporal self-attention operations, named spatio-temporal self-attention weighted VLAD (ST-SAWVLAD). In particular, sequential convolutional feature maps extracted from two modalities i.e., RGB and Flow are receptively fed into the self-attention module to learn soft spatio-temporal assignments parameters, which enabling aggregate not only detailed spatial information but also fine motion information from successive video frames. In experiments, we evaluate ST-SAWVLAD by using competitive action recognition datasets, UCF101 and HMDB51, the results shcoutstanding performance. The source code is available at:https://github.com/badstones/st-sawvlad.