Thanks for your reply. I didn’t use docker. I wonder if use torch.cuda.empty_cache after validation would help or not?
Besides, the memory is shown as follows:
          total        used        free      shared  buff/cache   available

Mem:         257662       41849      166048       44896       49764      169666
Swap:         16383        3492       12891