I faced a problem when i try to train my dataset on FCOS3D/MMdetection3D.

The error info:
RuntimeError: CUDA out of memory. Tried to allocate 2.83 GiB (GPU 6; 31.75 GiB total capacity; 28.25 GiB already allocated; 2.09 GiB free; 28.60 GiB reserved in total by PyTorch)
I have a question, why the used memory of CUDA:3 is twice as much as others? My guess is that there was a problem during one iteration that caused the memory to not be freed.

97E814D4-BFDF-4CCA-89E7-3C559888B25C645×590 17.7 KB