Teacher-student collaborative knowledge distillation for image classification

Applied Intelligence, vol. 53, pp. 1997–2009

Abstract

A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve model performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on knowledge distillation and self-distillation. The method consists of two parts: learning in the teacher network and self-teaching in the student network. Learning in the teacher network allows the student network to use knowledge from the teacher network. Self-teaching in the student network is to build a multi-exit network based on self-distillation and provide deep features as supervised information for training. In the inference stage, we use ensembles to vote on the classification results of multiple sub-models in the student network. The experimental results demonstrate the superior performance of our method compared with a traditional knowledge distillation method and a self-distillation-based multi-exit network.

Authors 6

1. Chuanyun Xu

   Chongqing Normal University · Chongqing University of Technology

   Affiliation as printed

   College of Computer and Information Science, Chongqing Normal University, Chongqing, 401331, China

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 400054, China

2. Wenjian Gao corresponding

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 400054, China

3. Tian Li Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Computer Science Department, RWTH Aachen University, Aachen, 52074, Germany

4. Nanlan Bai

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 400054, China

5. Gang Li

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 400054, China

6. Yang Zhang

   Chongqing Normal University

   Affiliation as printed

   College of Computer and Information Science, Chongqing Normal University, Chongqing, 401331, China