Thanks for the response!
You are correct, detect_anomaly more than doubles computation time. My first question was aiming at figuring out how this function could stabilize training, i.e. which settings are changed? Is there a documentation anywhere that describes the settings this function adjusts? This could help me find an intermediate solution without the whole bunch of overhead from detect_anomaly.
Could you explain to me what you mean exactly by storing each batch?