Read-All-in-Once (RAiO): Multi-layer Contextual Architecture for Long-Text Machine Reading Comprehensionopen access
- Authors
- Phan, Tuan-Anh; Jung, Jason J.; Bui, Khac-Hoai Nam
- Issue Date
- Jul-2023
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- Bit error rate; Computational modeling; Context modeling; Data mining; long-text machine reading comprehension; Natural language processing; Natural language processing; Question answering (information retrieval); question-answering system; Task analysis; Text processing; Transformers
- Citation
- IEEE Access, v.11, pp 77873 - 77879
- Pages
- 7
- Journal Title
- IEEE Access
- Volume
- 11
- Start Page
- 77873
- End Page
- 77879
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/67648
- DOI
- 10.1109/ACCESS.2023.3298100
- ISSN
- 2169-3536
- Abstract
- Machine reading comprehension (MRC) is a cutting-edge technology in natural language processing (NLP), which focus on teaching machines to read and understand the meaning of texts based on the emergence of large-scale datasets and neural network models. Recently, with the successful development of pre-trained transformer models (e.g., BERT), MRC has advanced significantly, surpassing human parity in several public datasets and being applied in various NLP tasks (e.g., QA systems). Nevertheless, long document MRC is still a remain challenge since the transformer-based models are limited by the input length. For instance, several well-known pre-trained language models such as BERT and RoBERTa are limited by 512 tokens. This study aims to provide a new simple approach for long document MRC. Specifically, recent state-of-the-art models follow the architecture with two crucial stages for reading long texts in order to enable local and global context representations. In this study, we present a new architecture that is able to enrich the global information of the context with one stage by exploiting the interaction of different levels of semantic units of the context (i.e., sentence and word level). Therefore, we name the proposed model as RAiO (Read-All-in-Once) approach. For the experiment, we evaluate RAiO on two benchmark long document MRC datasets such as NewsQA and NLQuAD. Accordingly, the experiment shows promising results of the proposed approach compared with strong baselines in this research field. Author
- Files in This Item
-
- Appears in
Collections - College of Software > School of Computer Science and Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.