Could you rerun the code using CUDA_LAUNCH_BLOCKING=1 python script.py args and post the debug messages again? Alternatively, you could also run your code on the CPU.
I’m not sure, if the printed shapes are really belonging to the problematic batch.