Basically when you first allocates cuda memory, if it is smaller than what we call a “block” (which I think is 256MB), we allocate a whole block of memory, and cache the rest.

I’m trying to see what you’re saying by measuring it and it doesn’t match up:
import torch 
import pynvml
pynvml = pynvml
pynvml.nvmlInit()
preload = 0
prev = 0
def nvml_used():
    handle = pynvml.nvmlDeviceGetHandleByIndex(torch.cuda.current_device())
    info   = pynvml.nvmlDeviceGetMemoryInfo(handle)
    return b2mb(info.used)
def b2mb(x): return int(x/2**20)
def consume_gpu_ram(n): return torch.ones((n, n)).cuda()
def consume_gpu_ram_64mb():  return consume_gpu_ram(2**12)
def consume_gpu_ram_256mb(): return consume_gpu_ram(2**13)
def mem(): 
    global preload, prev
    this = nvml_used()
    delta_cached = this - preload
    delta_used = this - prev
    prev = this
    print(f"   nvml used: {delta_used:4d}, allocated: {delta_cached:4d}")
    print(f"pytorch used: {b2mb(torch.cuda.memory_allocated()):4d}, allocated: {b2mb(torch.cuda.memory_cached()):4d}\n")

print("preloading:")
mem()
_ = torch.ones((1, 1)).cuda()
mem()
preload = nvml_used()

print("\nrunning:")
x1 = consume_gpu_ram_64mb()
mem()
x2 = consume_gpu_ram_256mb()
mem()
del x2
mem()
x3 = consume_gpu_ram_64mb()
mem()

So we have two parts to the program, the first one forces the loading of all that’s needed to run cuda, which allocates 0.5GB on the card and then it does a few allocations. And we measure torch.cuda’s allocated and cached and compare it to nvml numbers of allocated - I rigged the nvml_used() function to match torch.cuda.memory_cached()'s behavior by subtracting the memory allocated right after preloading happened, hence the different behavior during and after preloading.
nvml used reports don’t report the right thing once memory gets reported and caching kicks in. empty_cache needs to be called to get it right, but then it’d mess up with memory_cached repors which this code is exploring, so you can ignore nvml_used numbers.
Running it:
preloading:
   nvml used:   10, allocated:   10
pytorch used:    0, allocated:    0

   nvml used:  495, allocated:  505
pytorch used:    0, allocated:    1


running:
   nvml used:   64, allocated:   64
pytorch used:   64, allocated:   65

   nvml used:  256, allocated:  320
pytorch used:  320, allocated:  321

   nvml used:    0, allocated:  320
pytorch used:   64, allocated:  321

   nvml used:    0, allocated:  320
pytorch used:  128, allocated:  321

First, where is that block of cache you were referring to? When 0.5 GB is consumed pytorch reports 1MB of cache as compared to you saying that it should have allocated a block of about 256MB in cache.
Second, any reason why memory_allocated isn’t including those 0.5GB? Surely, for a user who isn’t going to dig into details this would appear very confusing. If she starts with a card not used by any process and allocates a tensor of 100MB, they would see it consuming 600MB, but reported only 100MB by pytorch.
Finally, from this report I understand that I didn’t original get what memory_cached() really is. memory_cached isn’t showing how much memory pytorch has cached that’s not currently used. It just says how much memory pytorch pre-allocated, but only some of it it may be free. So really to see how much available to use memory in cache is  memory_cached()-memory_allocated().
So my only remaining puzzle to understand is whether any of the 0.5GB allocated on the first cuda call is ever available to the user for allocating tensors, because currently it appears that when I use pytorch my card’s RAM is really TOTALRAM-0.5GB*num_of_processes. So If 2 processes use the card, I lose 1GB (!) of RAM.
Thank you.
edit: in case you decide to study the code do note that it has an issue of reporting delta used for nvml, but absolute used (allocated) for pytorch.cuda. So the first column (used) is somewhat bogus. while it’s easy to fix pytorch used to report delta, nvml can’t report the correct numbers because of caching here, so really just ignore the first column.