The Impact of LoRA on the Emergence of Clusters in Transformers

Code

• github.com/HugoKoubbi/Transformers-2024-LoRA
  Officialnone★ 2

Abstract

In this paper, we employ the mathematical framework on Transformers developed by sander2022sinkformers,geshkovski2023emergence,geshkovski2023mathematical to explore how variations in attention parameters and initial token values impact the structural dynamics of token clusters. Our analysis demonstrates that while the clusters within a modified attention matrix dynamics can exhibit significant divergence from the original over extended periods, they maintain close similarities over shorter intervals, depending on the parameter differences. This work contributes to the fine-tuning field through practical applications to the LoRA algorithm hu2021lora,peft, enhancing our understanding of the behavior of LoRA-enhanced Transformer models.

Reproductions