@ptrblck that’s correct. Without the log, the loss is no longer cross entropy and your p(x) * q(x) is computed instead.
The wrong loss function appears to work better with different learning rates as well as with or without momentum, but I haven’t extensively compared the two loss functions (e.g. I could try switching optimizers).
@googlebot Can you walk me through “you can deduce the consequences?” I’m not sure I understand how cross entropy relates to geometric and arithmetic means.
The task isn’t a typical supervised classification task, so it’s difficult to define what an easy sample is, but the task admits an exact Bayesian baseline and the wrong loss function does better than the correct loss function on datum requiring using inferred latent variables. We think of these datum as the “harder” cases.