I’m not sure, but I assume this shouldn’t be the case.
Are you seeing the large memory usage without using tensorboard?
However, you are also using detect_anomaly, which could slow down your code significantly.
Could you disable it and check the memory usage again?