LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models
<|reference_start|>LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models: In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs.   In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.<|reference_end|>
@article{guo2024lopt:,
    title={LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models},
    author={Shouchang Guo, Sonam Damani, Keng-hao Chang},
    journal={arXiv preprint arXiv:2406.19486},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.19486},
    primaryClass={cs.CL cs.AI cs.ET cs.LG eess.SP}
}