Could you try to collect the cuDNN log as @ngimel described here 26?
Also, could you run the script again with CUDA_LAUNCH_BLOCKING=1 python script.py args?
This is just a wild guess, but do you see your GPU(s) running out of memory?
While this might be counter-intuitive, since a larger batch size runs fine, cuDNN might try to use different algorithms (using less memory) for different batch sizes. I’m really not sure at this point, what might cause this issue so I’m just trying to guess what might happen.