Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks
<|reference_start|>Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks: The $\ell_{1,\infty}$ norm is an efficient structured projection but the complexity of the best algorithm is unfortunately $\mathcal{O}\big(n m \log(n m)\big)$ for a matrix in $\mathbb{R}^{n\times m}$. In this paper, we propose a new bi-level projection method for which we show that the time complexity for the $\ell_{1,\infty}$ norm is only $\mathcal{O}\big(n m \big)$ for a matrix in $\mathbb{R}^{n\times m}$, and $\mathcal{O}\big(n + m \big)$ with full parallel power. We generalize our method to tensors and we propose a new multi-level projection, having an induced decomposition that yields a linear parallel speedup up to an exponential speedup factor, resulting in a time complexity lower-bounded by the sum of the dimensions, instead of the product of the dimensions. we provide a large base of implementation of our framework for bi-level and tri-level (matrices and tensors) for various norms and provides also the parallel implementation. Experiments show that our projection is $2$ times faster than the actual fastest Euclidean algorithms while providing same accuracy and better sparsity in neural networks applications.<|reference_end|>
@article{perez2024multi-level,
    title={Multi-level projection with exponential parallel speedup; Application to
  sparse auto-encoders neural networks},
    author={Guillaume Perez and Michel Barlaud},
    journal={arXiv preprint arXiv:2405.02086},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.02086},
    primaryClass={cs.LG}
}