Hello, could you try the following,

Shift just the model to GPU and see an increase in memory,
Shift just one batch of data to the GPU and see the increase in memory,
Now shift both model as well as your data to GPU (Note that the Cuda driver occupies an extra of around 2GB(at least on my system, I do not whether its universally same or not))
Do not perform forward or backward pass.

It might be that your data is too large, or your model is large. If none is the issue then we would have to dig down further