In-Context Learning of Energy Functions
<|reference_start|>In-Context Learning of Energy Functions: In-context learning is a powerful capability of certain machine learning models that arguably underpins the success of today's frontier AI models. However, in-context learning is critically limited to settings where the in-context distribution of interest $p_{\theta}^{ICL}( x|\mathcal{D})$ can be straightforwardly expressed and/or parameterized by the model; for instance, language modeling relies on expressing the next-token distribution as a categorical distribution parameterized by the network's output logits. In this work, we present a more general form of in-context learning without such a limitation that we call \textit{in-context learning of energy functions}. The idea is to instead learn the unconstrained and arbitrary in-context energy function $E_{\theta}^{ICL}(x|\mathcal{D})$ corresponding to the in-context distribution $p_{\theta}^{ICL}(x|\mathcal{D})$. To do this, we use classic ideas from energy-based modeling. We provide preliminary evidence that our method empirically works on synthetic data. Interestingly, our work contributes (to the best of our knowledge) the first example of in-context learning where the input space and output space differ from one another, suggesting that in-context learning is a more-general capability than previously realized.<|reference_end|>
@article{schaeffer2024in-context,
    title={In-Context Learning of Energy Functions},
    author={Rylan Schaeffer, Mikail Khona, Sanmi Koyejo},
    journal={arXiv preprint arXiv:2406.12785},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12785},
    primaryClass={cs.LG}
}