Hi all,
I want to optimize only some variables. For example, create a Tensor, and then pass only a part of it to the optimizer. However, I am getting can't optimize a non-leaf Tensor error. Is there a way to do what I am trying to do?
I have attached a minimal example below. In this case, I want to optimize the source vector such that it matches the target, but I want only the middle 6 elements/parameters of the source vector to be optimized/change, and I don’t want to use masked loss to accomplish this.
import torch
import torch.optim as optim
import torch.nn as nn


target = torch.randn(10)
print('target: ', target)

# I want to optimize the source vector such that it matches the target
source = torch.Tensor(10).requires_grad_(True)
print('source: ', source)

# making sure source requires grad, while target doesn't
print('target req grad: ', target.requires_grad)
print('source req grad: ', source.requires_grad)

# passing only some parameters (middle 6, in this case) to the optimizer
optimizer = optim.SGD([source[2:-2]], lr=0.001)

criterion = nn.L1Loss()

starting_diff = criterion(target, source)
print('starting diff: ', starting_diff)
# actual optimization loop
while criterion(target, source) > 0.01:
    optimizer.zero_grad()
    loss = criterion(target, source)
    print('loss: ', loss)
    loss.backward()
    optimizer.step()