Transformers Handle Endogeneity in In-Context Linear Regression
<|reference_start|>Transformers Handle Endogeneity in In-Context Linear Regression: We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares $(\textsf{2SLS})$ solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the $\textsf{2SLS}$ method, in the presence of endogeneity.<|reference_end|>
@article{liang2024transformers,
    title={Transformers Handle Endogeneity in In-Context Linear Regression},
    author={Haodong Liang, Krishnakumar Balasubramanian, Lifeng Lai},
    journal={arXiv preprint arXiv:2410.01265},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01265},
    primaryClass={stat.ML cs.AI cs.LG econ.EM math.ST stat.TH}
}