Hello,
I’m having a strange issue where not a single one of my model’s parameters is getting updated. I saw another answer that’s similar to the problem I have (Model.parameters() is None while training 53), but the solution that’s given doesn’t quite solve my problem, because notably every one of my model’s parameter grads is None.
for p in model.parameters():
    if p.grad is not None:
        print(p.grad.data)

I.e. the above loop won’t print out anything for my model. And even after calling loss.backward() and optimizer.step(), the parameter grads all remain None.
I’ve verified that a loss is being calculated, it just seems that optimization isn’t taking place, even if the backward() and step() functions are definitely getting run.
This is my optimization code:
def step(batch, model, criterion, optimizer=None):
    # let go of old gradients
    model.zero_grad()
    
    X = batch["X"].to(DEVICE)
    y = batch["y"].to(DEVICE)

    ## Forward Pass ##
    predictions = model(inputs)

    ## Calculate Loss ##
    loss = criterion(predictions, y)

    if optimizer is not None:
        # backward pass + optimize
        loss.backward()
        optimizer.step()
        
    return loss

def train_model(model=None, lr=0.01):
    criterion = nn.CrossEntropyLoss().to(DEVICE)
    params = list(filter(lambda p: p.requires_grad, model.parameters()))
    optimizer = torch.optim.Adam(params=params, lr=lr)

    for epoch in range(1, N_EPOCHS+1):
        for i, batch in enumerate(tqdm(train_loader)):
            loss = step(batch, model, criterion, optimizer=optimizer)

model = NeuralNet()
model.to(DEVICE)
train_model(model=model)

The only thing I can think of is that it’s that how I’ve split my optimization code into functions, rather than keeping it all in the same scope. I was able to get a toy neural net with toy randn()-generated data to train properly in a single training loop that isn’t spread out across multiple functions (the model was initialized within the same scope it was trained).
I have a couple pre-trained torch.nn modules in NeuralNet(): a pre-trained Embedding layer and a pre-trained ResNet, both of which have frozen weights by setting requires_grad to False.
What could be causing this issue?