For Full-Text PDF, please login, if you are a member of IEICE,|
or go to Pay Per View on menu list, if you are a nonmember of IEICE.
Spatio-Temporal Self-Attention Weighted VLAD Neural Network for Action Recognition
Shilei CHENG Mei XIE Zheng MA Siqi LI Song GU Feng YANG
IEICE TRANSACTIONS on Information and Systems
Publication Date: 2021/01/01
Online ISSN: 1745-1361
Type of Manuscript: LETTER
Category: Biocybernetics, Neurocomputing
human action recognition, video representation, VLAD, self-attention module,
Full Text: PDF(511.9KB)>>
As characterizing videos simultaneously from spatial and temporal cues have been shown crucial for video processing, with the shortage of temporal information of soft assignment, the vector of locally aggregated descriptor (VLAD) should be considered as a suboptimal framework for learning the spatio-temporal video representation. With the development of attention mechanisms in natural language processing, in this work, we present a novel model with VLAD following spatio-temporal self-attention operations, named spatio-temporal self-attention weighted VLAD (ST-SAWVLAD). In particular, sequential convolutional feature maps extracted from two modalities i.e., RGB and Flow are receptively fed into the self-attention module to learn soft spatio-temporal assignments parameters, which enabling aggregate not only detailed spatial information but also fine motion information from successive video frames. In experiments, we evaluate ST-SAWVLAD by using competitive action recognition datasets, UCF101 and HMDB51, the results shcoutstanding performance. The source code is available at:https://github.com/badstones/st-sawvlad.