PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

arXiv (Cornell University)

Abstract

In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.

Authors 5

1. Zequan Liu Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Aachen , Germany

2. Yi Zhao

   University of Pennsylvania

   Affiliation as printed

   University of Pennsylvania , USA

3. Ming Tan

   Southern University of Science and Technology

   Affiliation as printed

   Southern University of Science and Technology , Shenzhen , China

4. Wei Zhu

   University of Hong Kong

   Affiliation as printed

   University of Hong Kong , Hong Kong , China

5. Aaron Xuxiang Tian

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University , Pittsburgh , USA