I’m trying to measure the memory usage of each layer by torch.cuda.max_memory_allocated() and torch.cuda.reset_max_memory_allocated():
for layer in sequential_module:
    torch.cuda.reset_max_memory_allocated(0)
    x = layer(x)
    size = torch.cuda.max_memory_allocated(0)
    sizes.append(size)

I understood torch.cuda.reset_max_memory_allocated() can be used to reset the starting point to track the maximum memory allocations. So I’ve expected that torch.cuda.max_memory_allocated() after reset always returns 0. But it doesn’t.
In [1]: import torch

In [2]: torch.cuda.max_memory_allocated(0)
Out[2]: 0

In [3]: x = torch.rand(100, 100, 100, device=0)

In [4]: torch.cuda.max_memory_allocated(0)
Out[4]: 4865536

In [5]: torch.cuda.reset_max_memory_allocated(0)

In [6]: torch.cuda.max_memory_allocated(0)
Out[6]: 4865536

How do I understand this behavior?