set_detect_anomaly should only be used for debugging, as it could significantly slow down your code besides the apparently observed increase in memory.
Once you’ve fixed all errors you should disable it. Also, your script should raise a warning claiming:
UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.