The learner-adjusted assessment tool for endotracheal intubation considering examiners’ different expectations of competence

Abstract

Objective: Endotracheal intubation (ETI) is the most reliable way to manage the airway. Stepwise deliberate practice and mastery training is essential in maintaining and promoting the skill of intubation. This study was conducted to identify differences in examiners’ expectations regarding competent skill performance and to develop learner-adjusted assessment tools with appropriate levels according to student and resident learners based on the expectations and limited observation of performance by examiners.Methods: This was an exploratory, psychometric study using a simple airway part task trainer. The draft ETI assessment tool from the literatures, previous tools, and the preliminary learner-adjusted assessment tool for students and residents were developed and analyzed. Knowledge-based and competence-based items for each learner group were identified based on experts’ expectations. The final learner-adjusted tools were refined through analyzing the content validity, internal consistency, and interrater reliability based on assessing the observed performance of 14 students and 12 residents by ten experts.Results: The preliminary student-adjusted assessment tool and resident-adjusted assessment tool had 12 items on the checklist with a ternary scoring system and a ternary scoring checklist including 15 items, and an overall Global Rating Scale.The final student-adjusted assessment tool was composed of a ternary scoring checklist including 9 items (total CVI, 86.6%; Cronbach’s α, 0.83; interrater reliability, 0.64). The resident-adjusted assessment, on the other hand, was also composed of a ternary scoring checklist including 12 items (total CVI, 86.4%; Cronbach’s α, 0.7; interrater reliability, 0.78), in addition to global rating scale including ‘rating of the overall process’.Conclusion: The experts had different expectations regarding the level of competence in each step according to learner groups with different levels of difficulty. Understanding the factors influencing assessments can provide a guide for teaching and objectively assessing to the examiner.