What is your batch size? I think its too high for your gpu to allocate to its memory. As I said use gradient accumulation to train your model.
If you want to train with batch size of desired_batch_size, then divide it by a reasonable number like 4 or 8 or 16…, this number is know as accumtulation_steps. Now change your batch size for the dataset to desired_batch_size/accumulation_steps and train your model as below
for epoch in range(epochs):
    for i, (inputs, labels) in enumerate(training_set):
        predictions = model(inputs)                     # Forward pass
        loss = loss_function(predictions, labels)       # Compute loss function
        loss = loss / accumulation_steps                # Normalize our loss (if averaged)
        loss.backward()                                 # Backward pass
        if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
            optimizer.step()                            # Now we can do an optimizer step
            model.zero_grad()                           # Reset gradients tensors