When Are Bias-Free ReLU Networks Like Linear Networks?
<|reference_start|>When Are Bias-Free ReLU Networks Like Linear Networks?: We investigate the expressivity and learning dynamics of bias-free ReLU networks. We firstly show that two-layer bias-free ReLU networks have limited expressivity: the only odd function two-layer bias-free ReLU networks can express is a linear one. We then show that, under symmetry conditions on the data, these networks have the same learning dynamics as linear networks. This allows us to give closed-form time-course solutions to certain two-layer bias-free ReLU networks, which has not been done for nonlinear networks outside the lazy learning regime. While deep bias-free ReLU networks are more expressive than their two-layer counterparts, they still share a number of similarities with deep linear networks. These similarities enable us to leverage insights from linear networks, leading to a novel understanding of bias-free ReLU networks. Overall, our results show that some properties established for bias-free ReLU networks arise due to equivalence to linear networks, and suggest that including bias or considering asymmetric data are avenues to engage with nonlinear behaviors.<|reference_end|>
@article{zhang2024when,
    title={When Are Bias-Free ReLU Networks Like Linear Networks?},
    author={Yedi Zhang, Andrew Saxe, Peter E. Latham},
    journal={arXiv preprint arXiv:2406.12615},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12615},
    primaryClass={cs.LG}
}