Conservation laws are considered to be fundamental laws of nature. It has broad application in many fields including physics, chemistry, biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to view numerical PDE solvers as a MDP and to use (deep) RL to learn new solvers. As a proof of concept, we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy. Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach. In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes. Our code is released anomynously at https://github.com/qwerlanksdf/L2D.
This paper proposes to use reinforcement learning for constructing discretziation stencils of numerical schemes. More specifically, the method focuses on the widely used WENO schemes, which are an established class of finite difference schemes. Within this context, the method aims for training models to infer the weighting for a specific stencil with eight flux terms.
For RL this task requires a continuous action space, and the DDPG algorithm is used for training the policy. The network itself is an MLP with 6 layers, and ca. 20000 weights in total. This is a significant number, given the focus on 1D problems.
The tests are quite thorough and interesting, while at the same time being limited in scope. The paper targets 1D cases, which make the problem very low-dimensional. Despite the simplicity, only a single data set (Burgers) is used, and a single modified target function with a u^4 term. Targeting 1D casesl, I would have expected a broader range of tests and model equations.
Despite the limited scope of the models, table 1 and 2 assess a nice range of different timestep and discretization parameters. I found it very interesting to see that the method consistently outperforms the regular WENO scheme. The gains are relatively small, with 4-5%, but WENO already represents a quite accurate scheme, so it's surely not easy to outperform it.
While reading the paper, I was wondering about the bigger picture, i.e. using RL in the context of discretization stencils. We have model equations, and discretized versions of all operators involved in training. Why employ a "brute force" approach like RL here? Wouldn't it be better in terms of efficiency and potentially also accuracy to train the stencils in a supervised manner, e.g., with a more accurate discretization as reference? One could argue that it would be expensive to pre-compute such data, but I think RL scales even worse to higher dimensional problems.
What's also missing in the current version is a more thorough discussion of inference and training performance. I guess that despite the small model problems, the training takes a substantial amount of time. And due to the large size of the trained model, which has to be evaluated for every single node in the 1D mesh, it's probably also quite slow. I think this is worth a discussion in the text. One could even estimate the number of operations necessary to evaluate the model, and run a higher-order WENO scheme for a "fair" comparison.
Minor, but in equation (1), I guess the t subscript should indicate a material derivative, and just just a time derivative, right? This could be clarified in the text (or written out).
I am somewhat on the edge with this paper - the 1D case for the two equations is carefully evaluated in the submission, and it's great to see the trained model can improve the accuracy across a fairly wide range of settings. As such, it's definitely a good and interesting first step. On the other hand, there are a range of open questions, as outlined above, and it's not clear whether the approach could be easily translated to higher dimensions. I hope the authors can clarify some of these points in the rebuttal, right now I'm leaning towards the positive side.