Same issue. In my case it happens only for certain batch sizes. I haven’t found a pattern though.
Would be great to get to the bottom of this. I am also circumventing the issue in the way @GlassyWing posted.