This is probably expected as different shapes can cause different algorithms/kernels to be run which will expectedly have numerical variations. If this is a problem and you are on an Ampere GPU, you can see if setting export NVIDIA_TF32_OVERRIDE=0 improves the relative difference or (as an extreme measure) using double precision improves things.