An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
<|reference_start|>An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings: There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. First, we compare SAMI to Direct Preference Optimization (DPO) on a multi-task benchmark (MT-Bench), using a stronger model to generate training data for a weaker one across diverse categories (humanities, STEM, extraction, coding, math, reasoning, and roleplay). Our results indicate that one iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories. Second, we examine SAMI's impact on mathematical accuracy (GSM-8K) relative to supervised fine-tuning (SFT). While SAMI increases zero-shot performance by 1.1%, SFT is more effective with a 3.2% boost. However, SAMI shows interesting scaling trends. When given 10 attempts, SAMI improves accuracy by 3.9%, while SFT achieves a 10.1% increase. Combining SAMI with SFT yields an additional improvement of 1.3% in multi-attempt settings, though single-attempt accuracy remains unchanged.<|reference_end|>
@article{govande2024an,
    title={An Exploration of Self-Supervised Mutual Information Alignment for
  Multi-Task Settings},
    author={Soham Govande},
    journal={arXiv preprint arXiv:2410.01704},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01704},
    primaryClass={cs.CL}
}