I want to optimize my init_tensor instead of optimizing the model. My data loader train_dataloader gives the initial value of init_tensor.
model.eval()
for inputs in train_dataloader:
     init_tensor = inputs["init_tensor"]
     optimizer = optim.SGD(init_tensor, learning_rate)
     output = model(init_tensor)
     loss = loss_fn(output)
     model.zero_grad()
     loss.backward()
     optimizer.step()



Defining the optimizer inside the training loop doesn’t look right. Will this hinder optimizing the init_tensor?


How can I move the optimizer initialization outside the training loop but still add init_tensor to optimizer?


Any help would be appreciated.