MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

arXiv (Cornell University)

Abstract

Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.

Authors 8

1. Pengjie Ren

   Shandong University

   Affiliation as printed

   Shandong University

2. Chengshun Shi

   Shandong University

   Affiliation as printed

   Shandong University

3. Shiguang Wu

   Shandong University

   Affiliation as printed

   Shandong University

4. Mengqi Zhang

   Shandong University

   Affiliation as printed

   Shandong University

5. Zhaochun Ren Aachen

   Leiden University

   Affiliation as printed

   Leiden University

6. Maarten de Rijke

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam

7. Zhumin Chen

   Shandong University

   Affiliation as printed

   Shandong University

8. Jiahuan Pei

   Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica