When can transformers compositionally generalize in-context?
<|reference_start|>When can transformers compositionally generalize in-context?: Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during training. Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components? Here we study a modular multitask setting that allows us to precisely control compositional structure in the data generation process. We present evidence that transformers learning in-context struggle to generalize compositionally on this task despite being in principle expressive enough to do so. Compositional generalization becomes possible only when introducing a bottleneck that enforces an explicit separation between task inference and task execution.<|reference_end|>
@article{kobayashi2024when,
    title={When can transformers compositionally generalize in-context?},
    author={Seijin Kobayashi, Simon Schug, Yassir Akram, Florian Redhardt,
  Johannes von Oswald, Razvan Pascanu, Guillaume Lajoie, Jo~ao Sacramento},
    journal={arXiv preprint arXiv:2407.12275},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12275},
    primaryClass={cs.LG cs.NE}
}