Hi.
When you create a tensor or create a model (that create tensors that represent your parameters), you allocate memory in your RAM (i.e your CPU memory). If you want your model to run in GPU then you have to copy and allocate memory in your GPU-RAM space. Note that, the GPU can only access the GPU-memory.
Pytorch by default stores everything in CPU (in fact torch tensors are wrappers over numpy objects) and you can call .cuda() or .to_device() to move a tensor to gpu. Example:
import torch
import torch.nn as nn
a=torch.zeros((10,10)) #in cpu
a=a.cuda() #copy the CPU memory to GPU memory

class mymodel(nn.module):
         ....
model=mymodel().cuda() #moves all the parameters from the model to cuda

To store memory in cuda you typically allocate CPU-ram memory and then use a function called cudaMemcpy link 25 to move between GPU and CPU. That is way you need to tell PyTorch to move things to GPU (so it call cudaMemcpy under the hood, well not exactly, see the last paragraph).
There is no standard way to tell PyTorch to move everything to GPU (as far as I now). Just take in account that a GPU can  only access GPU memory space (which is divided in Global, shared and registers). I think there are special cases with the pinned memory but, as a global, you can think it in that way (GPU only access GPU). Then to run in GPU the data must be in GPU.
[As additional information regarding memory and performance] In other softwares like Theano the advice was to store as many amount of data in the GPU as you could, in order to avoid memory transfers through the PCI of your system (think that if you store your dataset on the GPU you avoid sending a batch each time you want to update your model). However, PyTorch overcomes this limitation by an efficient memory managment which is based in memory pools. These are memory managers. There are different possibilities. Theano used the cnmem module.  Memory managers are very used in practice. Malloc C function does something similar to avoid calling the OS each time you require a chunk of memory.
Hope it helps.