weizuo:

is it possible that some memory is allocated in the process, but not managed by the PT allocator (e.g., directly goes into the cuda allocator?)


Yes, this would be possible if you are using a custom CUDA extension and are allocating memory directly there.
Could you try to post a minimal, executable code snippet which would reproduce this behavior on a 40GB GPU?