Development and Evaluation of a Dual-Expertise, Utterance-Level Framework for LLM-Based Science Classroom Discourse Analysis

Abstract

This study proposes a novel coding framework for analyzing science classroom discourse using large language models (LLMs), adopting fine-grained utterance-level chunking aligned with the analytical units of LLMs to address limitations of global, lesson-level observation tools. Authentic middle school science classroom discourse was annotated through a dual-expertise and iterative process integrating the theoretical knowledge of science education faculty with the experiential insights of in-service teachers, supported by systematic rater training to ensure conceptual alignment and interpretive consistency at the utterance level. Through this process, a science education glossary comprising 137 instructional terms organized into 20 thematic categories was developed using a primarily bottom-up approach informed by established observation frameworks. Building on this theory-informed foundation, we systematically examined LLM-based methods for predicting instructional themes and quality, comparing structured prompting strategies with domain-adaptive fine-tuning across model architectures. These contributions lay a foundation for future research on interpretable, scalable, and pedagogically meaningful automated formative feedback to support teachers’ self-reflection and professional growth.