I set the flag to False and the error level is now reasonable. Never knew PyTorch on Ampere GPUs has this behaviour. Guess in most cases tf32 should be enough. Most networks have all kinds of regularizations anyway. Thank you!