stas:

Are you talking about the context of loading cuda, the first time it’s used? I get a pretty consistent 0.5GB per process with cuda100.


Yes. Each cuda context is fixed size. For PyTorch, we use something called primary ctx, which is unique per process.
The rest of 0.5GB comes from the caching allocator. Basically when you first allocates cuda memory, if it is smaller than what we call a “block” (which I think is 256MB), we allocate a whole block of memory, and cache the rest. Also, using random sampling methods for the 1st time also allocates additional cuda rng state, but that is also one time and fixed size.
Edit: The above info is not accurate. See follow up below.