Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization
<|reference_start|>Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization: We present PSiLON Net, an MLP architecture that uses $L_1$ weight normalization for each weight vector and shares the length parameter across the layer. The 1-path-norm provides a bound for the Lipschitz constant of a neural network and reflects on its generalizability, and we show how PSiLON Net's design drastically simplifies the 1-path-norm, while providing an inductive bias towards efficient learning and near-sparse parameters. We propose a pruning method to achieve exact sparsity in the final stages of training, if desired. To exploit the inductive bias of residual networks, we present a simplified residual block, leveraging concatenated ReLU activations. For networks constructed with such blocks, we prove that considering only a subset of possible paths in the 1-path-norm is sufficient to bound the Lipschitz constant. Using the 1-path-norm and this improved bound as regularizers, we conduct experiments in the small data regime using overparameterized PSiLON Nets and PSiLON ResNets, demonstrating reliable optimization and strong performance.<|reference_end|>
@article{biswas2024hidden,
    title={Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm
  Regularization},
    author={Aditya Biswas},
    journal={arXiv preprint arXiv:2404.19112},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.19112},
    primaryClass={cs.LG stat.ML}
}