@tom Thank you for your in-time reply!
I can understand why the reserved space is getting larger increasingly. But from  after forward 1st epoch | memory_reserved: 5014MB, why the cached memory can’t be released automatically, and I have to use torch.cuda.empty_cache() to clean it. As I know, the caching allocator which is like Python reference will automatically release the cached space while there is no reference to the variable.
Besides, I have another question. The memory_summary of after forward 1st epoch  is listed below. Why the Non-releasable memory is generated during the forward of the net? Thanks!
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  476675 KB |    5012 MB |    5526 MB |    5061 MB |
|       from large pool |  476672 KB |    5012 MB |    5526 MB |    5061 MB |
|       from small pool |       3 KB |       0 MB |       0 MB |       0 MB |
|---------------------------------------------------------------------------|
| Active memory         |  476675 KB |    5012 MB |    5526 MB |    5061 MB |
|       from large pool |  476672 KB |    5012 MB |    5526 MB |    5061 MB |
|       from small pool |       3 KB |       0 MB |       0 MB |       0 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    5014 MB |    5014 MB |    5014 MB |       0 B  |
|       from large pool |    5012 MB |    5012 MB |    5012 MB |       0 B  |
|       from small pool |       2 MB |       2 MB |       2 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |    1178 MB |    1178 MB |    1521 MB |  351331 KB |
|       from large pool |    1176 MB |    1176 MB |    1519 MB |  351232 KB |
|       from small pool |       1 MB |       1 MB |       2 MB |      99 KB |
|---------------------------------------------------------------------------|
| Allocations           |       4    |      17    |     160    |     156    |
|       from large pool |       2    |       4    |       6    |       4    |
|       from small pool |       2    |      16    |     154    |     152    |
|---------------------------------------------------------------------------|
| Active allocs         |       4    |      17    |     160    |     156    |
|       from large pool |       2    |       4    |       6    |       4    |
|       from small pool |       2    |      16    |     154    |     152    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       5    |       5    |       5    |       0    |
|       from large pool |       4    |       4    |       4    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       2    |       3    |      30    |      28    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |       1    |       3    |      29    |      28    |
|===========================================================================|