Combining domain knowledge models with neural models has been challenging. End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train. In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems. The models are interpretable, extrapolative, dataefficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise. We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification. Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity. Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.
The authors present a new hybrid models that incorporates traditional models with neural networks by boosting and reducing the residuals in stages. They show 3 different boosting schemes: sequential boosting, parallel boosting and cyclical boosting (inspired by variational autoencoders). Furthermore the authors present the new Ljung-Box (LJB) loss function which reduces autocorrelation. They test on 2 simulated systems and on a physical DC Motor and put an emphasis on testing their result on extrapolated (unseen) data.
Strengths:
- The authors combine traditional models with deep learning approaches.
- The models can extrapolate to unseen data better than conventional deep learning approaches.
- The results show that the authors found a loss function that can reduce autocorrelation.
Weaknesses: 
- The paper lacks an exact explanation for every model that was used. This is made worse by a lack of consistency, e.g. the model Physics-Boost-Dense doesn't explain which of the 3 boosting schemes is used.
- The authors propose a new Ljung-Box (LJB)  loss function. It contains hyper parameter L "which should be larger than possible correlations". It is not mentioned how to find such a value or what the value for the experiments for. They show that this loss function can reduce autocorrelation, but it comes at the expense of higher RMSE. It is not clear in what scenario this is useful.
- The methodology is difficult to follow. E.g. there is a mathematical explanation in 3.1, but not for 3.2 or 3.3. In Fig. 1(1) and Fig. 1(2), they incorporate a traditional model, but not in Fig. 1(3). Later they show results for a cyclic method that incorporates a physical model, but it's not clear how that was done.
- The authors claim that they are using "a new method for creating testing data". However, testing on extrapolating data is not a new method.
Additional Comments: 
- Equation (2): I assume the hat should only be above f, not the entire f(x). If not, I don't see this symbol explained or used anywhere else in the paper.
- h_t in equation (3) and (4) are not explained in the text
- Fig. 1(1) and Fig. 1(3) are not mentioned in the text
- Figure 3 and 4 are hard to understand. Instead of State 0/1/2, it should say angle / angular velocity / acceleration. There should be a table to accompany this figure, so that it's easier to compare the models. As is, it's very hard to compare e.g. Physics-Ensemble-Dense with Dense-Ensemble-RNN.
- There are plenty of typos and a general lack of clarity. E.g. this sentence: "The  alternative  of  performing  end-to-end  of  all  the  models  at  once  is  an  significant  alternative."
- Add reference for original Ljung-Box test