I am new to pytorch but I do program some in CUDA (enough to be dangerous).
My understanding is that when, say,  a numpy array is passed as a tensor, then it is in GPU memory. It can then be manipulated by the GPU and finally transferred back to the CPU memory.
Then, there are CUDA tensors. CUDA also operates on GPU memory.

What is the difference?
Can GPU profiling be done in pytorch as it can be done in CUDA with the nvidia profiler?

Thank You
Tom