Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception. In this paper, we identify a fundamental issue of the standard MBRL framework -what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t. the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task. In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of the one-step ahead prediction is not always correlated with downstream control performance. This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed. We propose an initial method to mitigate the mismatch issue by reweighting dynamics model training. Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.
This paper discusses the old problem of mismatch between the ultimate reward obtained after optimizing a  decision (planning or control) over a probabilistic model (of dynamics) and  the training  objective for the model (log-likelihood). Experiments highlight that the NLL and reward can be very poorly correlated, that improvements in NLL initially improve reward but can later degrade it, and that models with similar NLLs can lead to very different rewards. A  reweighting trick is proposed and summarily evaluated.
I like the topic of this paper but there are several aspects which I see as making it weaker than my acceptance threshold.
First, the paper overclaims in originality. This mismatch problem is not new, it is an instance of a more general issue that end-to-end training and meta-learning try to address, and has been already studied in the context of MBRL by many authors, who actually proposed more substantial solutions. When I read the abstract I had the impression that the paper actually had a theoretical analysis showing the correlation problem, but there is no such thing, only experiments. Section 3 does not actually provide a new insight. Still, the experiments are interesting in that they reveal that the magnitude of the mismatch is probably more serious than most RL researchers believed.
Second, the 'fix' proposed is not well justified nor well tested (e.g. no quantiative comparisons, no comparisons against existing alternative methods to address the same problem, etc). This seriously weakens conclusions like "shows improvements in sample efficiency".
One concern I have about the experiments of fig 3 is that NLL can be really bad, thus distorting rho, which is not a robust measure. So I would only look at NLLs of models with good NLLs, to obtain a more interesting analysis.
Another concern about experiments is that I am not convinced that they were performed with SOTA MBRL methods and hyper-parameters (as demonstrated by SOTA performance on known benchmarks). Otherwise I could easily imagine how the mismatch could be much more severe than in the actual scenarios of interest.
Minor points:
Bottom of page 6 refers to visualizations but I did not see if or where they were shown.
Why the e in the numerator of eq 2e? Seems useless to put any constant there.
The section on 'Shaping the cost or reward' was not clear enough to me (please expand).