Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19: Predictive Algorithm Modeling and Interpretation Studyopen access
- Authors
- Baek, Sangwon; Jeong, Yeon Joo; Kim, Yun-Hyeon; Kim, Jin Young; Kim, Jin Hwan; Kim, Eun Young; Lim, Jae-Kwang; Kim, Jungok; Kim, Zero; Kim, Kyunga; Chung, Myung Jin
- Issue Date
- Jan-2024
- Publisher
- JMIR PUBLICATIONS, INC
- Keywords
- COVID-19; prognosis; prognostic; prognostics; prediction model; early triaging; interpretability; machine learning; predict; prediction; predictive; triage; triaging; emergency; severity; biomarker; biomarkers; SHAP; Shapley; clustering; hospital admission; hospital admissions; hospitalize; hospitalization; hospitalizations; neural network; neural networks; deep learning; Omicron; SARS-CoV-2; coronavirus
- Citation
- JOURNAL OF MEDICAL INTERNET RESEARCH, v.26
- Journal Title
- JOURNAL OF MEDICAL INTERNET RESEARCH
- Volume
- 26
- URI
- https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/90891
- DOI
- 10.2196/52134
- ISSN
- 1439-4456
1438-8871
- Abstract
- Background: Robust and accurate prediction of severity for patients with COVID-19 is crucial for patient triaging decisions. Many proposed models were prone to either high bias risk or low-to-moderate discrimination. Some also suffered from a lack of clinical interpretability and were developed based on early pandemic period data. Hence, there has been a compelling need for advancements in prediction models for better clinical applicability. Objective: The primary objective of this study was to develop and validate a machine learning-based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers. Methods: We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods. Results: Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910). Conclusions: RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - ETC > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/90891)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.