Multiple-Stage Knowledge Distillation

Applied Sciences, vol. 12, pp. 9453

Abstract

Knowledge distillation (KD) is a method in which a teacher network guides the learning of a student network, thereby resulting in an improvement in the performance of the student network. Recent research in this area has concentrated on developing effective definitions of knowledge and efficient methods of knowledge transfer while ignoring the learning ability of the student network. To fully utilize this potential learning ability and improve learning efficiency, this study proposes a multiple-stage KD (MSKD) method that allows students to learn the knowledge delivered by the teacher network in multiple stages. The student network in this method consists of a multi-exit architecture, and the students imitate the output of the teacher network at each exit. The final classification by the student network is achieved through ensemble learning. However, because this results in an unreasonable gap between the number of parameters in the student branch network and those in the teacher branch network, as well as a mismatch in learning capacity between these two networks, we extend the MSKD method to a one-to-one multiple-stage KD method. The experimental results reveal that the proposed method applied to the CIFAR100 and Tiny ImageNet datasets exhibits good performance gain. The proposed method of enhancing KD by changing the style of student learning provides new insight into KD.

Authors 7

1. Chuanyun Xu

   Chongqing Normal University · Chongqing University of Technology

   Affiliation as printed

   College of Computer and Information Science, Chongqing Normal University, Chongqing 401331, China

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing 400054, China

2. Nanlan Bai corresponding

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing 400054, China

3. Wenjian Gao

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing 400054, China

4. Tian Li Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Computer Science Department, RWTH Aachen University, 52074 Aachen, Germany

5. Mengwei Li

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing 400054, China

6. Gang Li corresponding

   Chongqing University of Technology

   Affiliation as printed

   School of Artificial Intelligence, Chongqing University of Technology, Chongqing 400054, China

7. Yang Zhang

   Chongqing Normal University

   Affiliation as printed

   College of Computer and Information Science, Chongqing Normal University, Chongqing 401331, China