TEMPORAL DIFFERENCE VARIATIONAL AUTO-ENCODER
To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.
There are several ingredients in this paper that I really liked. For example, (1) the notion that an agent should build a deterministic function of the past which implicitly captures the belief (the uncertainty or probability distribution about the state), by opposition for example to sampling trajectories to capture uncertainty, (2) modelling the world's dynamic in a learned encoded state-space (by opposition to the sensor space), (3) instead of modeling next-step probabilities p(z(t+1)|z(t)), model 'jumpy transitions' p(z(t+delta)|z(t)) to avoid unrolling at the finest time scale.
Now for the weak points:
(a) the justification for the training loss was not completely clear to me, although I can see that it has a variational flavor
(b) there is no discussion of the issue that we can't get a straightforward decomposition of the joint probability over the data sequence according to next-step probabilities via the chain rule of probabilities, so we don't have a clear way to compare the TD-VAE models with jumpy predictions against other more traditional models
(c) none of the experiments make comparisons against previously published models and quantitative results (admittedly because of (b) this may not be easy).
So I believe that the authors are onto a great direction of investigation, but the execution of the paper could be improved.