Fast Iterative Solver For Neural Network Method: I 1D Diffusion Problems
<|reference_start|>Fast Iterative Solver For Neural Network Method: I 1D Diffusion Problems: The discretization of the deep Ritz method [18] for the Poisson equation leads to a high-dimensional non-convex minimization problem, that is difficult and expensive to solve numerically. In this paper, we consider the shallow Ritz approximation to one-dimensional diffusion problems and introduce an effective and efficient iterative method, a damped block Newton (dBN) method, for solving the resulting non-convex minimization problem.   The method employs the block Gauss-Seidel method as an outer iteration by dividing the parameters of a shallow neural network into the linear parameters (the weights and bias of the output layer) and the non-linear parameters (the weights and bias of the hidden layer). Per each outer iteration, the linear and the non-linear parameters are updated by exact inversion and one step of a damped Newton method, respectively. Inverses of the coefficient matrix and the Hessian matrix are tridiagonal and diagonal, respectively, and hence the cost of each dBN iteration is $\mathcal{O}(n)$. To move the breakpoints (the non-linear parameters) more efficiently, we propose an adaptive damped block Newton (AdBN) method by combining the dBN with the adaptive neuron enhancement (ANE) method [25]. Numerical examples demonstrate the ability of dBN and AdBN not only to move the breakpoints quickly and efficiently but also to achieve a nearly optimal order of convergence for AdBN. These iterative solvers are capable of outperforming BFGS for select examples.<|reference_end|>
@article{cai2024fast,
    title={Fast Iterative Solver For Neural Network Method: I. 1D Diffusion
  Problems},
    author={Zhiqiang Cai, Anastassia Doktorova, Robert D. Falgout, C'esar Herrera},
    journal={arXiv preprint arXiv:2404.17750},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.17750},
    primaryClass={math.NA cs.NA}
}