During the forward pass intermediate tensors will be stored (output activations), which are needed for the gradient calculation during the backward pass.
Depending on the model architecture these activations might use more memory than the model parameters.
You could check the peak memory via torch.cuda.max_memory_allocated() using a batch size of 1.
Calling torch.cuda.empty_cache() will not avoid the OOM memory issue, but will instead only slow down your code.