This can happen when the cost of transferring data between RAM and GPU memory is more than the speedup of parallel computation on the GPU.
One case, as mentioned in @ptrblck’s answer is when your model is quite small. Another case is when you have too many back-and-forth transfers of data in your forward() function. See https://pytorch.org/docs/stable/notes/cuda.html 391 for an example of when data is transferred between RAM and GPU memory.