Video Search Reranking with Relevance Feedback Using Visual and Textual Similarities

Takamasa FUJII  Soh YOSHIDA  Mitsuji MUNEYASU  

Publication
IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences   Vol.E102-A   No.12   pp.1900-1909
Publication Date: 2019/12/01
Online ISSN: 1745-1337
DOI: 10.1587/transfun.E102.A.1900
Type of Manuscript: Special Section PAPER (Special Section on Smart Multimedia & Communication Systems)
Category: Multimedia Environment Technology
Keyword: 
video search,  reranking,  tag-based retrieval,  

Full Text: PDF(2.8MB)>>
Buy this Article




Summary: 
In video search reranking, in addition to the well-known semantic gap, the intent gap, which is the gap between the representation of the users' demand and the real search intention, is becoming a major problem restricting the improvement of reranking performance. To address this problem, we propose video search reranking based on a semantic representation by multiple tags. In the proposed method, we use relevance feedback, which the user can interact with by specifying some example videos from the initial search results. We apply the relevance feedback to reduce the gap between the real intent of the users and the video search results. In addition, we focus on the fact that multiple tags are used to represent video contents. By vectorizing multiple tags associated with videos on the basis of the Word2Vec algorithm and calculating the centroid of the tag vector as a collective representation, we can evaluate the semantic similarity between videos by using tag features. We conduct experiments on the YouTube-8M dataset, and the results show that our reranking approach is effective and efficient.