RTKD: Responsive Teacher Knowledge Distillation with heterogeneous students

Abstract

Knowledge Distillation (KD) is a learning methodology whereby a teacher model effectively transfers its knowledge to a student model. This KD process is often likened to a classroom scenario where an experienced and knowledgeable teacher transfers knowledge to students. However, this simple analogy falls short of capturing the dynamic and complex interactions observed between teachers and students of various levels in real-world classroom settings. In this paper, we propose Responsive Teacher Knowledge Distillation (RTKD), a novel KD method inspired by observed processes in actual classrooms. RTKD is a novel KD method wherein the teacher model self-improves by comprehensively incorporating learning outcomes from student models with various levels. This approach aims for richer and more adaptive knowledge transfer by enabling the teacher model to update itself by collecting and synthesizing learning feedback from various student models. We experimentally demonstrate the efficacy of RTKD through quantitative evaluations on various tasks requiring commonsense reasoning capabilities. Additionally, we validated the proposed model by conducting ablation studies on key hyperparameters such as temperature settings, high-capacity student model layer configurations, and the student contribution ratio. Further, the architectural validity of RTKD was experimentally confirmed through quantitative performance comparisons against several alternative architectures.