Modality shifting attention network for multi-modal video question answering
- Authors
- Kim, J.; Ma, M.; Pham, T.; Kim, K.; Yoo, C.D.
- Issue Date
- Jun-2020
- Publisher
- IEEE Computer Society
- Citation
- Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 10103 - 10112
- Pages
- 10
- Journal Title
- Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
- Start Page
- 10103
- End Page
- 10112
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/63389
- DOI
- 10.1109/CVPR42600.2020.01012
- ISSN
- 1063-6919
- Abstract
- This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities. MSAN is able to place importance weight on the two modalities for each sub-task using a component referred to as Modality Importance Modulation (MIM). Experimental results show that MSAN outperforms previous state-of-the-art by achieving 71.13% test accuracy on TVQA benchmark dataset. Extensive ablation studies and qualitative analysis are conducted to validate various components of the network.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Software > Department of Artificial Intelligence > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/63389)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.