hey, i’m back with this issue of setting only certain elements of the tensor “learnable” -
my net is more complicated than this example but maybe it’s enough to find my issue without describing the whole thing:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(2, 6, 3) #notice two channel input
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(16 * 6 * 6, 2)
        self.embed = torch.randn((1,10,10),requires_grad=True) # lets say the input is a 10x10 image, and so is the embedding before masking
        mask = torch.zeros((1,10,10),requires_grad=False)
        mask[0,:,5] = 1 # we want to mask the embedding such that only one row is actually updated during backprop
        self.embed = self.embed * mask # notice that embed requires_grad and mask doesn't require. is this how it should be done or vice versa?

    def forward(self, x):
        x = torch.cat([self.embed,x])
        out = self.conv1(x)
        out = flatten(out)
        return self.fc1(out)

should this work in theory?
in my case it’s slightly more complicated, because certain inputs belong to one category and i want to learn an embedding of each category, so the forward pass accepts x and category,it looks like this:
self.embed =  self.embed.repeat(category_num,1,1)
def forward(self,x, category):
     x = torch.cat([self.embed[category], x])
     out = self.conv1(x)
     out = flatten(out)
     return self.fc1(out)

i’m doing almost exactly that and getting a weird error that i couldn’t solve -
runtimeerror: trying to backward through the graph a second time, but the saved intermediate results have already been freed. specify retain_graph=true when calling backward the first time.