Sorry, my bad, I didn’t look correctly. In the first pass, it looked okay, but in the second pass the problem occurred – probably because the Adam params are zero in the first pass.
Using two separate optimizers like shown in the update gist sounds like the best idea in this case