LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
Abstract
Low-Rank Adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning foundation models by re-parameterizing the original matrix into the product of two low-rank matrices. Despite its efficiency, LoRA often yields inferior performance compared to full fine-tuning. In this paper, we propose LoRA-Pro to bridge this performance gap.
Firstly, we delve into the optimization processes in LoRA and full fine-tuning. We reveal that while LoRA employs low-rank approximation, it neglects to approximate the optimization process of full fine-tuning. To address this, we introduce a novel concept called the "equivalent gradient." This virtual gradient makes the optimization process on the re-parameterized matrix equivalent to LoRA, which can be used to quantify the differences between LoRA and full fine-tuning. The equivalent gradient is derived from the gradients of matrices and . To narrow the performance gap, our approach minimizes the differences between the equivalent gradient and the gradient obtained from full fine-tuning during the optimization process. By solving this objective, we derive optimal closed-form solutions for updating matrices and . Our method constrains the optimization process, shrinking the performance gap between LoRA and full fine-tuning.
Extensive experiments on natural language processing tasks validate the effectiveness of our method.
1 Introduction
Foundational models [Radford et al., 2021, Brown et al., 2020, Achiam et al., 2023, Kirillov et al., 2023, Rombach et al., 2022] have become the cornerstone of modern deep learning. By undergoing pre-training on massive datasets, these models typically exhibit excellent generalization and versatility. Remarkably, some foundation models even demonstrate emergent properties [Hoffmann et al., 2022, Kaplan et al., 2020]. As a result, foundation models have been widely applied to various downstream applications.
Despite these advantages, the huge number of parameters in foundational models hinders their broader application. The substantial parameter count results in high fine-tuning costs for these tasks. To address this issue, recent research has focused on parameter-efficient fine-tuning (PEFT) methods [Hu et al., 2022, Houlsby et al., 2019, Lester et al., 2021, Zhou et al., 2022]. PEFT methods reduce the fine-tuning cost by keeping the foundation models frozen and only fine-tuning small, additional lightweight adapters. With the majority of parameters frozen, PEFT enables faster fine-tuning and requires fewer computational resources.
Low-rank adaptation [Hu et al., 2022], also known as LoRA, is one of the most famous PEFT methods, which has been widely adopted across various domains. Inspired by previous works [Aghajanyan et al., 2021, Li et al., 2018], LoRA hypothesizes that the changes in weights during model adaptation exhibit a low-rank structure. To capture this, LoRA re-parameterizes these changes by expressing them as the product of two low-rank matrices: , where is a scaling factor, and and are low-rank matrices with rank . LoRA reduces the number of trainable parameters from to , thereby decreasing the cost of fine-tuning. However, despite its efficiency, LoRA’s fine-tuning performance often falls short compared to full fine-tuning [Hu et al., 2022, Liu et al., 2024, Ding et al., 2023].
In this paper, we propose a novel PEFT method, LoRA-Pro, aimed at bridging the gap between LoRA and full fine-tuning. While LoRA employs low-rank approximation by re-parametrizing weight changes as the product of two low-rank matrices, it falls short in approximating the optimization process of full fine-tuning. To measure their discrepancy in the optimization process, we propose a novel concept, “Equivalent Gradient", for LoRA optimization. Equivalent gradient characterizes the gradient of the original matrix after low-rank approximation (despite it not being directly trainable), is composed of gradients from matrices A and B. Thus, during LoRA fine-tuning, our goal is not only to approximate the matrix with low-rank matrices but also to minimize the difference between the equivalent gradient and the gradient from full fine-tuning during the gradient descent process. This is achieved by selecting appropriate gradients for matrices A and B, ensuring a more accurate and effective fine-tuning process. To achieve this, we formulate it as an optimization problem. We then derive theoretical solutions for the problem, presenting optimal gradients for updating matrices A and B. These solutions ensure that the equivalent gradient closely match the optimization dynamics of full fine-tuning. By doing so, we enhance the effectiveness LoRA, bridging the gap between LoRA and full fine-tuning.
Our main contributions are summarized as follows:
-
•
We identify that LoRA approximates low-rank matrices but neglects to approximate the optimization process of full parameter fine-tuning. This shortcoming is one of the reasons for the performance gap between LoRA and full fine-tuning.
-
•
We introduce the concept of Equivalent Gradient, which allows us to quantify the discrepancy in the optimization process between LoRA and full fine-tuning. By minimizing this discrepancy, we derive the optimal closed-form updated solutions for LoRA.
-
•
Extensive experiments on natural language processing tasks validate the effectiveness of our method.
2 Related Work
Parameter-Efficient Fine-Tuning. Given the huge size of foundation models, recent research has focused on developing parameter-efficient fine-tuning methods [Hu et al., 2022, Liu et al., 2024, Ding et al., 2023, Houlsby et al., 2019, Liu et al., 2023, Lester et al., 2021]. These methods aim to reduce the cost of fine-tuning by adjusting only a small portion of the model’s parameters. Generally, these methods fall into two main categories. The first category is adapter tuning [Houlsby et al., 2019, Sung et al., 2022, He et al., 2021, Zhang et al., 2024, Bapna and Firat, 2019, Hu et al., 2022], which involves inserting small neural network modules, called adapters, into specific layers of the model. During fine-tuning, we keep the model frozen and only fine-tune the lightweight adapter modules, significantly reducing the memory footprint for fine-tuning. The second category is prompt tuning [Lester et al., 2021, Zhou et al., 2022, Li and Liang, 2021, Liu et al., 2022]. Prompt tuning adapts the models to specific tasks by adding specially designed prompts or learnable tokens to the input data, rather than directly modifying the internal parameters of foundation models. In this paper, we focus on LoRA [Hu et al., 2022], a prominent method within the realm of adapter tuning.
Low Rank Adaptation. Low-rank adaptation, initially referred to as LoRA [Hu et al., 2022], has evolved into a broad category encompassing parameter-efficient fine-tuning methods based on low-rank approximations [Hu et al., 2022, Liu et al., 2024, Hayou et al., 2024, Kalajdzievski, 2023, Zhang et al., 2023, Kopiczko et al., 2024, Hyeon-Woo et al., 2022, Zhang and Pilanci, 2024, Wang et al., 2024, Zhao et al., 2024]. LoRA [Hu et al., 2022] assumes that the changes in the weights of pre-trained models exhibit a low-rank structure. Consequently, it re-parameterizes these changes as the product of low-rank matrices, thereby reducing the cost associated with fine-tuning.
Several variants of LoRA have been proposed to address different aspects of this approach. For example, DoRA [Liu et al., 2024] improves LoRA [Hu et al., 2022] by incorporating a learnable magnitude vector to re-scale the normalized product of low-rank matrices. Another variant, rsLoRA Kalajdzievski [2023], introduces a new scaling factor to stabilize training in high-rank scenarios. LoRA+[Hayou et al., 2024] improves upon LoRA by applying different learning rates to the two low-rank matrices. Additionally, Galore [Zhao et al., 2024] employs SVD to project the gradients of full parameter training into a low-rank space, thereby reducing the memory footprint during pre-training and fine-tuning.
3 Method
In this section, we begin by revisiting LoRA [Hu et al., 2022] in Section 3.1. Following this, we conduct a comparison between LoRA and full fine-tuning from an optimization perspective in Section 3.2. Finally, in Section 3.3, we point out that LoRA falls short in approximating full fine-tuning during the optimization process, and we introduce LoRA-Pro as a solution to bridge this performance gap.
3.1 Revisit Low Rank Adaptation
First of all, let’s dive back into Low-Rank Adaptation (LoRA) [Hu et al., 2022]. LoRA’s core idea revolves around recognizing the low-rank structure of the change matrix in the standard fine-tuning process. This insight allows LoRA [Hu et al., 2022] to re-parameterize the change matrix into the product of two low-rank matrices,
(1) |
Here, represents the pre-trained weight matrix, and are the low-rank matrices, and is a scaling factor. For LoRA [Hu et al., 2022], , while for rsLoRA [Kalajdzievski, 2023], . Here, is the hyper-parameter and denotes the rank. Consequently, LoRA significantly reduces the number of fine-tuning parameters from to .
3.2 LoRA v.s. Full Fine-tuning
Despite widespread applications across various domains, LoRA’s performance still falls short when compared to full fine-tuning. In this part, we review and compare LoRA and full fine-tuning in the optimization process. In full fine-tuning, we utilize differential to analyze the relationship between changes in the loss and changes in the weights:
(2) |
where and denotes the changes of the parameter and the loss , and is the Frobenius norm. To minimize the loss function, we typically set (omitting the learning rate for simplicity), which results in .
In LoRA optimization, given that , we compute the differential using the chain rule:
(3) | ||||
Similarly, LoRA sets and , and thus . Moreover, employing the chain rule, we derive:
(4) |
3.3 Low-Rank Adaptation with Equivalent Gradient
In this section, Equivalent Gradient. From Equation (3), we can see that changes in matrices and are inherently linked to changes in matrix through the chain rule:
(6) |
In comparison to full fine-tuning, this is equivalent to updating using the gradient . This critical relationship has been neglected in the LoRA optimization process. Hence, we hypothesize that by carefully adjusting the gradients of matrices and in such a way that under LoRA closely approximates the gradient from full fine-tuning, we can effectively bridge the gap between LoRA and full fine-tuning.
Based on this relationship, we define the concept of equivalent gradient in Definition 3.3. Equivalent gradient describes the gradient of the matrix following low-rank adaptation, despite not being a trainable parameter. To narrow the performance gap, our goal is to carefully select suitable and to minimize the distance between the equivalent gradient and the gradient under full fine-tuning . Hence, our objective is:
(7) | ||||
s.t. | ||||
Closed-form Solution. Fortunately, Equation (7) admits a closed-form solution. According to Theorem 3.3, we obtain the optimal gradients for matrices and , ensuring that the equivalent gradient achieves the best approximation to the full fine-tuning gradient. Moreover, we observe that and can be expressed as and , respectively, indicating that we do not explicitly possess the full fine-tuning gradient . Therefore, our approach involves back-propagating in standard LoRA and adjusting the gradients of matrices and using the closed-form solution outlined in Theorem 3.3.
Although Theorem 3.3 provides a closed-form solution to the optimization problem , this does not necessarily mean that updating matrices and with this solution will decrease the loss. To address this, we have Theorem 3.3, which guarantees a decrease in the loss during the optimization process. This theorem indicates that the change in loss, , can be expressed as a negative scalar multiplied by the sum of two positive definite quadratic forms. This relationship ensures that during the update process, thus consistently driving the optimization process towards a lower loss.
Selection of X. Although the equivalent gradient itself is not directly related to the matrix , the presence of plays a significant role in the updates of matrices and . We select an appropriate such that and remain close to and respectively. Consequently, we minimize their Frobenius norm, as demonstrated in Equation (41). In practical terms, and do not share common eigenvalues. Therefore, according to Theorem 3.3, we can determine a unique optimal for updating matrices and .
5 Experimental Results
In this section, we evaluate our LoRA-Pro method across various natural language understanding datasets. To provide a comprehensive comparison, we include several baseline methods: 1) full fine-tuning and the standard LoRA [Hu et al., 2022]. 2) LoRA variants maintaining the original structure, such as rsLoRA [Kalajdzievski, 2023], LoRA+ [Hayou et al., 2024], PiSSA [Meng et al., 2024], 3) oRA variants with modified structures, including DoRA [Liu et al., 2024] and AdaLoRA [Zhang et al., 2023].
The results are shown in Table 1. We fine-tune the T5-base model [Raffel et al., 2020] with the baseline methods on a subset of GLUE datasets. From Table 1, we observe that LoRA-Pro achieves the highest scores on 3 out of 5 datasets and the highest average score across all 5 datasets. Moreover, on average over 5 datasets, LoRA-Pro suppass standard LoRA [Hu et al., 2022] with a margin of 6.72. These results validate the effectiveness of our methods.
Method | MNLI | SST2 | CoLA | QNLI | MRPC | Avg. |
Full FT | 86.33±0.00 | 94.75±0.21 | 80.70±0.24 | 93.19±0.22 | 84.56±0.73 | 87.91 |
LoRA | 85.30±0.04 | 94.04±0.11 | 69.35±0.05 | 92.96±0.09 | 68.38±0.01 | 82.08 |
PiSSA | 85.75±0.07 | 94.07±0.06 | 74.27±0.39 | 93.15±0.14 | 76.31±0.51 | 84.71 |
rsLoRA | 85.73±0.10 | 94.19±0.23 | 72.32±1.12 | 93.12±0.09 | 52.86±2.27 | 79.64 |
LoRA+ | 85.81±0.09 | 93.85±0.24 | 77.53±0.20 | 93.14±0.03 | 74.43±1.39 | 84.95 |
DoRA | 85.67±0.09 | 94.04±0.53 | 72.04±0.94 | 93.04±0.06 | 68.08±0.51 | 82.57 |
AdaLoRA | 85.45±0.11 | 93.69±0.20 | 69.16±0.24 | 91.66±0.05 | 68.14±0.28 | 81.62 |
LoRA-GA | 85.70±0.09 | 94.11±0.18 | 80.57±0.20 | 93.18±0.06 | 85.29±0.24 | 87.77 |
LoRA-Pro | 86.92±0.08 | 94.46±0.24 | 82.25±1.01 | 92.89±0.12 | 87.50±0.65 | 88.80 |
6 Conclusion
In this paper, we introduce LoRA-Pro, a novel approach designed to bridge the performance gap between LoRA and full fine-tuning. To bridge the performance gap, we introduce the concept of Equivalent Gradient, which allows us to quantify the difference in the optimization process between LoRA and full fine-tuning. By minimizing this discrepancy, we derive the optimal closed-form updated solutions for LoRA. Moreover, we prove that the solutions guarantee the loss decease during optimization. These solutions not only apply a low-rank approximation to the fine-tuning matrix but also maintain consistency with the optimization of full fine-tuning, enabling more effective fine-tuning. Finally, we validate the effectiveness of our method through extensive experiments on natural language processing tasks.
References
- Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- Aghajanyan et al. [2021] A. Aghajanyan, S. Gupta, and L. Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In ACL-IJCNLP, 2021.
- Bapna and Firat [2019] A. Bapna and O. Firat. Simple, scalable adaptation for neural machine translation. In EMNLP-IJCNLP, 2019.
- Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
- Ding et al. [2023] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023.
- Hayou et al. [2024] S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024.
- He et al. [2021] R. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng, J. Low, L. Bing, and L. Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In ACL-IJCNLP, 2021.
- Hoffmann et al. [2022] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. In NeurIPS, 2022.
- Houlsby et al. [2019] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
- Hu et al. [2022] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
- Hyeon-Woo et al. [2022] N. Hyeon-Woo, M. Ye-Bin, and T.-H. Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. In ICLR, 2022.
- Kalajdzievski [2023] D. Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023.
- Kaplan et al. [2020] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
- Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In ICCV, 2023.
- Kopiczko et al. [2024] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation. In ICLR, 2024.
- Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
- Li et al. [2018] C. Li, H. Farkhoor, R. Liu, and J. Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018.
- Li and Liang [2021] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, 2021.
- Liu et al. [2023] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
- Liu et al. [2024] S.-y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024.
- Liu et al. [2022] X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022.
- Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
- Meng et al. [2024] F. Meng, Z. Wang, and M. Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948, 2024.
- Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
- Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
- Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- Sung et al. [2022] Y.-L. Sung, J. Cho, and M. Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In CVPR, 2022.
- Sutskever et al. [2013] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.
- Wang et al. [2024] S. Wang, L. Yu, and J. Li. Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint arXiv:2407.05000, 2024.
- Zhang and Pilanci [2024] F. Zhang and M. Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. In ICML, 2024.
- Zhang et al. [2023] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, 2023.
- Zhang et al. [2024] R. Zhang, J. Han, C. Liu, A. Zhou, P. Lu, Y. Qiao, H. Li, and P. Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR, 2024.
- Zhao et al. [2024] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-efficient llm training by gradient low-rank projection. In ICML, 2024.
- Zhou et al. [2022] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
————Appendix————
The structure of Appendix is as follows,
Appendix A Notations
In Table 2, we detail the notations utilized in our paper.
Notation | Description |
---|---|
scaling factor in lora | |
, | low rank matrices in LoRA |
gradients of full rank fine-tuning | |
gradients of matrix A in lora | |
gradients of matrix B in lora | |
differential of the loss function | |
differential of the matrix A | |
differential of the matrix B | |
Frobenius Norm | |
Frobenius inner product |
Appendix B Proof of Theoretical Results
B.1 Proof of Theorem 3.3
B.2 Proof of Theorem 3.3
B.3 Proof of Theorem 3.3
Appendix C Optimization Algorithms
In this section, we present the pseudo-codes for implementing our LoRA-Pro method using the SGD [Sutskever et al., 2013] and AdamW [Loshchilov and Hutter, 2019] optimizers. These are detailed in Algorithm 1 and Algorithm 2, respectively.
In the standard SGD algorithm, as illustrated in Algorithm 1, all we need to do is adjusting the gradients of matrices and with the solutions in Theorem 3.3.
In AdamW optimizer, the implementation becomes more complex. Several modifications are necessary. Firstly, in order to mimic full fine-tuning, after adjusting the gradients of matrices and , we need to compute the equivalent gradient,
(45) |
Subsequently, we calculate the first and second moments of this equivalent gradient to derive the corresponding AdamW gradient, . Secondly, we determine the gradients with respect to matrices and as follows:
(46) |
Thirdly, the weight decay process must be adjusted. In line with full fine-tuning, the weight decay is given by:
(47) |
This can be decomposed into:
(48) |