한국어 말하기 평가에서 나타난 채점자 신뢰도 및 일관성 연구A Study on Raters’ Reliability and Consistency Observed in Korean Speaking Tests
- Authors
- 조윤정; 양명희
- Issue Date
- 2018
- Publisher
- 한국화법학회
- Keywords
- 말하기 수행 평가; 신뢰도; 채점자 간 신뢰도; 채점자 내 신뢰도; 일관성; 엄격성; 채점자 훈련; speaking performance test; reliability; inter-rater reliability; intra-rater reliability; consistency; severity; rater training
- Citation
- 화법연구, no.40, pp 105 - 128
- Pages
- 24
- Journal Title
- 화법연구
- Number
- 40
- Start Page
- 105
- End Page
- 128
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/2861
- DOI
- 10.18625/jsc.2018..40.105
- ISSN
- 1598-9542
- Abstract
- This study aims to examine the reliability and consistency of raters of Korean speaking tests. This basic research can identify the type of education that can enhance the reliability of rating in speaking tests, which are mainly assessed using subjective assessment criteria. In this study, we intended to study the rating tendencies, reliability, and consistency of raters by conducting two separate experiments under the same conditions and separated by a certain interval. As a result of the analysis using the FACETS program based on the Many-Facets Rasch Measurement model, the second rating saw an overall improvement in inter-rater reliability; however, raters’ consistency varied regardless of their career experience in the field of Korean language education. It is necessary to train raters to improve assessment reliability, and these study results confirmed that individualized training that can be customized for each rater’s personality or characteristics is needed. In addition, this could be an alternative to training raters if the intention is to improve self-consistency through self-observation by using scientific tools that can measure the rater’s reliability and consistency.
This study aims to examine the reliability and consistency of raters of Korean speaking tests. This basic research can identify the type of education that can enhance the reliability of rating in speaking tests, which are mainly assessed using subjective assessment criteria. In this study, we intended to study the rating tendencies, reliability, and consistency of raters by conducting two separate experiments under the same conditions and separated by a certain interval. As a result of the analysis using the FACETS program based on the Many-Facets Rasch Measurement model, the second rating saw an overall improvement in inter-rater reliability; however, raters’ consistency varied regardless of their career experience in the field of Korean language education. It is necessary to train raters to improve assessment reliability, and these study results confirmed that individualized training that can be customized for each rater’s personality or characteristics is needed. In addition, this could be an alternative to training raters if the intention is to improve self-consistency through self-observation by using scientific tools that can measure the rater’s reliability and consistency.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Humanities > ETC > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/2861)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.