SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings
<|reference_start|>SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings: Soft prompt tuning techniques have recently gained traction as an effective strategy for the parameter-efficient tuning of pretrained language models, particularly minimizing the required adjustment of model parameters. Despite their growing use, achieving optimal tuning with soft prompts, especially for smaller datasets, remains a substantial challenge. This study makes two contributions in this domain: (i) we introduce SuperPos-Prompt, a new reparameterization technique employing the superposition of multiple pretrained vocabulary embeddings to improve the learning of soft prompts. Our experiments across several GLUE and SuperGLUE benchmarks consistently highlight SuperPos-Prompt's superiority over Residual Prompt tuning, exhibiting an average score increase of $+6.4$ in T5-Small and $+5.0$ in T5-Base along with a faster convergence. Remarkably, SuperPos-Prompt occasionally outperforms even full fine-tuning methods. (ii) Additionally, we demonstrate enhanced performance and rapid convergence by omitting dropouts from the frozen network, yielding consistent improvements across various scenarios and tuning methods.<|reference_end|>
@article{sadraeijavaeri2024superpos-prompt:,
    title={SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with
  Superposition of Multi Token Embeddings},
    author={MohammadAli SadraeiJavaeri, Ehsaneddin Asgari, Alice Carolyn McHardy,
  Hamid Reza Rabiee},
    journal={arXiv preprint arXiv:2406.05279},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05279},
    primaryClass={cs.CL cs.AI}
}