Skimming through your code, I couldn’t find anything obviously wrong.
Could your issue be related to this one 118?
To investigate it further, could you tell which OS, GPU (if used), CUDA + cuDNN version you are using?