A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

neural information processing systems, pp. 59765–59797

Abstract

Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.

Authors 4

1. Qiang Huang

   Harbin Institute of Technology

   Affiliation as printed

   Harbin Institute of Technology (Shenzhen)

2. Xinyan Xiao

   Baidu (China)

   Affiliation as printed

   Baidu

3. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University

4. Jun Yu

   Hangzhou Dianzi University

   Affiliation as printed

   Hangzhou Dianzi University