Here is the summary printed within pytorch, which looks smaller. But OOM is triggered as long as the usage in nvidia-smi hits the limit
In [15]: print(tc.cuda.memory_summary())
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |   27378 KB |   36136 KB |   52001 KB |   24622 KB |
|       from large pool |   27345 KB |   36106 KB |   51732 KB |   24386 KB |
|       from small pool |      33 KB |      49 KB |     269 KB |     236 KB |
|---------------------------------------------------------------------------|
| Active memory         |   27378 KB |   36136 KB |   52001 KB |   24622 KB |
|       from large pool |   27345 KB |   36106 KB |   51732 KB |   24386 KB |
|       from small pool |      33 KB |      49 KB |     269 KB |     236 KB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   43008 KB |   43008 KB |   43008 KB |       0 B  |
|       from large pool |   40960 KB |   40960 KB |   40960 KB |       0 B  |
|       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |   15629 KB |   18617 KB |   59813 KB |   44184 KB |
|       from large pool |   13614 KB |   16573 KB |   57533 KB |   43919 KB |
|       from small pool |    2015 KB |    2044 KB |    2280 KB |     265 KB |
|---------------------------------------------------------------------------|
| Allocations           |      17    |      24    |     241    |     224    |
|       from large pool |       7    |       9    |      13    |       6    |
|       from small pool |      10    |      17    |     228    |     218    |
|---------------------------------------------------------------------------|
| Active allocs         |      17    |      24    |     241    |     224    |
|       from large pool |       7    |       9    |      13    |       6    |
|       from small pool |      10    |      17    |     228    |     218    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       3    |       3    |       3    |       0    |
|       from large pool |       2    |       2    |       2    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       6    |       8    |     105    |      99    |
|       from large pool |       3    |       3    |       8    |       5    |
|       from small pool |       3    |       5    |      97    |      94    |
|===========================================================================|