Development and validation of a transformer model-based early warning score for real-time prediction of adverse outcomes in the emergency department

Abstract

This study aimed to develop and validate a transformer-based early warning score (TEWS) system for predicting adverse events (AEs) in the emergency department (ED). We conducted a retrospective study analyzing adult ED visits at a tertiary hospital. The TEWS was developed to predict five AEs within 24 h: vasopressor use, respiratory support, intensive care unit admission, septic shock, and cardiac arrest. Performance was evaluated and compared using the area under the receiver operating characteristic curve (AUROC) and bootstrap-based t-test. External validation was performed using the Marketplace for Medical Information in Intensive Care (MIMIC)-IV-ED database. Transfer learning was applied using 1% and 5% of the external data. A total of 414,748 patients was analyzed in the development cohort (AEs, 3.7%), and 410,880 patients (AEs, 6.7%) were included in the external validation cohort. Compared to the modified early warning score (MEWS), the TEWS incorporating 13 variables and the vital signs-only TEWS demonstrated superior prognostic performance across all AEs. The AUROC ranged from 0.833 to 0.936 for TEWS and 0.688 to 0.874 for MEWS. In external validation, the TEWS also showed acceptable discrimination with AUROC values of 0.759 to 0.905. Transfer learning significantly improved the performance, increasing AUROC values to 0.846-0.911. The TEWS system was successfully integrated into the electronic health record (EHR) system of the study hospital, providing real-time risk assessment for ED patients. We developed and validated an artificial intelligence-based early warning score system that predicts multiple adverse outcomes in the ED and was successfully integrated into the EHR system.