Feature Estimations Based Correlation Distillation for Incremental Image Retrieval

IEEE Transactions on Multimedia, vol. 24, pp. 1844–1856

Abstract

Deep learning for fine-grained image retrieval in an incremental context is less investigated. In this paper, we explore this task to realize the model’s continuous retrieval ability. That means, the model enables to perform well on new incoming data and reduce forgetting of the knowledge learned on preceding old tasks. For this purpose, we distill semantic correlations knowledge among the representations extracted from the new data only so as to regularize the parameters updates using the teacher-student framework. In particular, for the case of learning multiple tasks sequentially, aside from the correlations distilled from the penultimate model, we estimate the representations for all prior models and further their semantic correlations by using the representations extracted from the new data. To this end, the estimated correlations are used as an additional regularization and further prevent catastrophic forgetting over all previous tasks, and it is unnecessary to save the stream of models trained on these tasks. Extensive experiments demonstrate that the proposed method performs favorably for retaining performance on the already-trained old tasks and achieving good accuracy on the current task when new data are added at once or sequentially.

Authors 6

1. Wei Chen Aachen Leiden Institute of Advanced Computer Science Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

   Computer Science, Universiteit Leiden Leiden Institute of Advanced Computer Science, 382026 Leiden, Zuid-Holland, Netherlands, (e-mail: w.chen@liacs.leidenuniv.nl)

2. Yu Liu

   Dalian University of Technology

   Affiliation as printed

   DUT-RU International School of Information Science and Engineering, Dalian University of Technology, Dalian, China

3. Nan Pu Aachen Leiden Institute of Advanced Computer Science [Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

   [Science, Leiden University Leiden Institute of Advanced Computer Science, 382026 Leiden, Zuid-Holland, Netherlands, (e-mail: n.pu@liacs.leidenuniv.nl)]

4. Weiping Wang

   National University of Defense Technology

   Affiliation as printed

   College of Systems Engineering, NUDT, Changsha, China

5. Li Liu

   National University of Defense Technology · University of Oulu

   Affiliation as printed

   College of Systems Engineering, NUDT, Changsha, China

   Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland; College of Systems Engineering, NUDT, Changsha, China

6. Michael S. Lew Aachen Leiden Institute of Advanced Computer Science [Media Lab

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

   [Media Lab, Universiteit Leiden Leiden Institute of Advanced Computer Science, 382026 Leiden, Zuid-Holland, Netherlands, 2333CA (e-mail: m.s.lew@liacs.leidenuniv.nl)]