I am trying to extract the trainable parameters of a pre-trained model and then train them over a custom loss function. However, the gradients in this case are None and hence despite the opt.step() there is no update on the model.parameters().

lstTxtOnly = 'text dataset'
model = torch.load('infersent.allnli.pickle')

theta_0 = []
for p in model.parameters():
    theta_0.append(p.data)

max_epochs = 5
learning_rate = 1e-2

for epoch in range(max_epochs+1):

    running_loss = 0.0
    R = torch.nn.functional.softmax(pre_R)
    z_pred = torch.max(R, 1)[1]
    
    X = model.encode(lstTxtOnly, bsize=128, tokenize=False, verbose=False)
    
    # Losses
    custom_loss = 'normal kmeans loss'
    frob_norm_squared = 0
    for (p1,p2) in zip(list(model.parameters()), theta_0):
        frob_norm_squared += torch.sum(torch.abs(p1.data - p2) **2)
    regularizer = frob_norm_squared

    # Optimizer
    opt_F= torch.optim.Adam(model.parameters(), lr = learning_rate)
    
    # zero the parameter gradients
    opt_F.zero_grad()   
    
    loss = custom_loss  + regularizer  
    loss.backward(retain_graph=True)

    for param in model.parameters():
        print(param.grad)
    
    opt_F.step()

I have already checked some other posts 1 where one tries to update the model parameters manually 2 but in my case the gradient itself is zero.
If I remove the retain_graph=True argument the code breaks. Can you please advice how to debug this issue?