Performance of large language models in non-English medical ethics-related multiple choice questions: comparison of ChatGPT performance across versions and languages

Abstract

BackgroundAs large language models (LLMs) evolve, assessing their competence in ethically sensitive domains such as medical ethics has become increasingly important. Since medical ethics is a universal component of medical education, disparities in AI performance across languages may result in unequal benefits for learners. Therefore, it is essential to examine performances in non-English contexts. While previous studies have evaluated performance of Chat Generative Pre-trained Transformer(ChatGPT) on English-language multiple-choice questions (MCQs) in medical ethics, none have examined version-based improvements across non-English contexts. This study therefore evaluated ChatGPT versions 3.5, 4.0, and 4.5 for MCQs on Korean medical ethics and their English translations, with a focus on performance trends across versions and languages.MethodsWe selected 36 MCQs from the Korean National Medical Licensing Examination and the Comprehensive Clinical Medicine Evaluation databases. Each question was entered ten times per ChatGPT version (3.5, 4.0, 4.5) and language (Korean, English) for a total of 60 trials. Additionally, to assess the model's capacity to identify the ethical core without relying on the options provided, 31 of the 36 questions were modified by masking the correct choice. Accuracy was analyzed using independent sample t-tests and Mann Whitney U test, and consistency was assessed using Krippendorff's alpha.ResultsOverall, the accuracy and consistency of ChatGPT improved with each version. Version 4.5 achieved near-perfect scores and high reliability in both languages, while version 3.5 showed limited performance, particularly in the Korean test. Performance gaps between languages decreased with model upgrades but remained statistically significant in version 4.5 for some questions. In the masked-answer condition, all versions showed notable drops in accuracy and consistency, with version 4.5 still outperforming earlier versions. However, the performance remained below 50%, indicating limitations in the model's autonomous ethical reasoning.ConclusionsChatGPT demonstrated substantial improvements in medical ethics MCQ performance across versions, particularly in terms of consistency and accuracy. However, performance disparities between languages and reduced accuracy under masked answer conditions highlight the ongoing limitations of non-English ethical reasoning and context recognition. These findings emphasize the need for further research on language-sensitive fine-tuning and the evaluation of LLMs in specialized ethical domains. The findings suggest that advanced LLMs may serve as valuable supplementary tools in medical education and clinical ethics training. At the same time, the observed language disparities call for context-sensitive adaptations to prevent inequities in practice.