Exploratory study on the potential of ChatGPT as a rater of second language writing
- Authors
- Shin, Dongkwang; Lee, Jang Ho
- Issue Date
- Jun-2024
- Publisher
- SPRINGER
- Keywords
- Automated writing evaluation; ChatGPT; Many-faceted Rasch model; Rater evaluation
- Citation
- EDUCATION AND INFORMATION TECHNOLOGIES
- Journal Title
- EDUCATION AND INFORMATION TECHNOLOGIES
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/74549
- DOI
- 10.1007/s10639-024-12817-6
- ISSN
- 1360-2357
1573-7608
- Abstract
- In recent years, various strategies have been employed to integrate ChatGPT into the field of second language (L2) teaching and learning. In line with such efforts, this study investigates the potential of ChatGPT as an automated writing evaluation (AWE) tool for L2 assessment, given the lack of systematic and quantitative investigation into human ratings and GPT-based scoring chatbot's ratings. We took an innovative approach by utilising ChatGPT's new feature called 'My GPTs', which is a customised chatbot builder based on GPT-4. The dataset for assessment consisted of 50 English essays written by Korean secondary-level EFL students, which were rated by the developed GPT-based scoring chatbot and two in-service English teachers. The intraclass correlation coefficient results suggested a strong similarity between human rater and ChatGPT scores. However, those based on the multifaceted Rasch model further revealed that ChatGPT showed a slightly greater deviation from the model than its human counterparts. This study demonstrates the potential of ChatGPT in AWE, providing an accessible and supplementary tool to L2 teachers' ratings.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Education > Department of English Education > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.