Thanks for the detailed information!
I think I was a bit mislead by the strange behavior and assumed some invalid inputs/updates/running estimates etc. Based on your description it seems the training setup is just creating this issue. I’m not sure if it’s possible or worth “debugging” it, as you already found some workarounds.