Investigating the Effectiveness of HyperTuning via Gisting
<|reference_start|>Investigating the Effectiveness of HyperTuning via Gisting: Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-based hypernetworks. We introduce HyperLlama, a set of Gisting-based hypernetworks built on Llama-2 models that generates task-specific soft prefixes based on few-shot inputs. In experiments across P3, Super-NaturalInstructions and Symbol Tuning datasets, we show that HyperLlama models can effectively compress information from few-shot examples into soft prefixes. However, they still underperform multi-task fine-tuned language models with full attention over few-shot in-context examples. We also show that HyperLlama-generated soft prefixes can serve as better initializations for further prefix tuning. Overall, Gisting-based hypernetworks are economical and easy to implement, but have mixed empirical performance.<|reference_end|>
@article{phang2024investigating,
    title={Investigating the Effectiveness of HyperTuning via Gisting},
    author={Jason Phang},
    journal={arXiv preprint arXiv:2402.16817},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.16817},
    primaryClass={cs.CL}
}