Visual representation learning using graph-based higher-order heuristic distillation for cell detection in blood smear images

Abstract

Background and objective: In many real-world scenarios, including the blood smear domain, it is difficult for detection networks to achieve good performance because image annotation is usually time consuming and expensive. To address this issue, similarity-based distillation (SD) methods, considered the soft version of contrastive learning, are applied to learn a better visual representation without requiring any supervision of the downstream task. Motivated by our theoretical analysis, we treat standard SD methods as the maximization of common 1-hop neighboring key points between two queries in an attributed graph, where nodes represent query and key data points. However, such first-order graph heuristic methods that calculate the likelihood of an unseen link between target nodes by using up to 1-hop neighborhoods are normally limited by insufficient representation power and even lack of generalization ability. Methods: Therefore, in this paper, we propose a novel higher-order heuristic distillation (H2D) method that distills knowledge about more general and powerful higher-order heuristic features based on a more than 1-hop relationship in the attributed graph. To do this, we utilize a graph neural network model to learn the higher-order heuristic features on the attributed graph constructed by query and key data representations and transfer the knowledge from the teacher to the student encoder. Results: Our method outperforms the previous state-of-the-art SD methods in the cell detection task on the blood smear dataset as well on open databases (Pascal VOC and MS COCO). Conclusions: Our proposed model allow teacher encoder to transfer the knowledge about more general and powerful higher-order heuristic embeddings to the student and enables better learning for visual representation on cell detection task using blood smear images.