Note that besides the parameters and input, the training will also use memory for the forward activations, which are needed to compute the gradients (which also need to be stored on the device).
Is your training working with a smaller batch size?