Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment
<|reference_start|>Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment: Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.<|reference_end|>
@article{wilcoxson2024polynomial,
    title={Polynomial Regression as a Task for Understanding In-context Learning
  Through Finetuning and Alignment},
    author={Max Wilcoxson, Morten Svendg{aa}rd, Ria Doshi, Dylan Davis, Reya Vir,
  Anant Sahai},
    journal={arXiv preprint arXiv:2407.19346},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19346},
    primaryClass={cs.LG cs.CL}
}