miner_tom:

My understanding is that when, say, a numpy array is passed as a tensor, then it is in GPU memory.


No, numpy arrays will share the same underlying data, if tensor = torch.from_numpy(array) is used (no copy triggered) and can be pushed to the GPU via tensor = tensor.to('cuda'), where operations on this tensor will be executed on the GPU.



 miner_tom:

Then, there are CUDA tensors.


I’m not sure what CUDA tensors are. If you are referring to PyTorch tensors on the GPU, then note that each tensor has a device associated with it and can be stored on different devices, such as the CPU, GPU, TPU, etc.



 miner_tom:

Can GPU profiling be done in pytorch as it can be done in CUDA with the nvidia profiler?


Yes, you can use Nsight systems/compute, pyprof, the autograd.profiler 1 etc.