A Simple Framework for Scene Graph Reasoning with Semantic Understanding of Complex Sentence Structureopen access
- Authors
- Heo, Yoonseok; Kang, Sangwoo
- Issue Date
- Sep-2023
- Publisher
- MDPI
- Keywords
- multimodal deep learning; scene graph reasoning; multimodal transformer; multi-task learning
- Citation
- MATHEMATICS, v.11, no.17
- Journal Title
- MATHEMATICS
- Volume
- 11
- Number
- 17
- URI
- https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/89133
- DOI
- 10.3390/math11173751
- ISSN
- 2227-7390
2227-7390
- Abstract
- A rapidly expanding multimedia environment in recent years has led to an explosive increase in demand for multimodality that can communicate with humans in various ways. Even though the convergence of vision and language intelligence has shed light on the remarkable success over the last few years, there is still a caveat: it is unknown whether they truly understand the semantics of the image. More specifically, how they correctly capture relationships between objects represented within the image is still regarded as a black box. In order to testify whether such relationships are well understood, this work mainly focuses on the Graph-structured visual Question Answering (GQA) task which evaluates the understanding of an image by reasoning a scene graph describing the structural characteristics of an image in the form of natural language together with the image. Unlike the existing approaches that have been accompanied by an additional encoder for scene graphs, we propose a simple yet effective framework using pre-trained multimodal transformers for scene graph reasoning. Inspired by the fact that a scene graph can be regarded as a set of sentences describing two related objects with a relationship, we fuse them into the framework separately from the question. In addition, we propose a multi-task learning method that utilizes evaluating the grammatical validity of questions as an auxiliary task to better understand a question with complex structures. This utilizes the semantic role labels of the question to randomly shuffle the sentence structure of the question. We have conducted extensive experiments to evaluate the effectiveness in terms of task capabilities, ablation studies, and generalization.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - IT융합대학 > 소프트웨어학과 > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/89133)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.