Hi, I’m having problem training network of the following structure (structure inspired by a recent paper):
Given input X, Y, we want to first find a hidden variable based on X:
X → fc1 → fc2 → hidden
Then we want to slice the hidden variable so that we can assign them as weights/bias to fc3, fc4
And then we compute:
Y → fc3 → fc4 → prediction
We have a ground truth to compute loss with prediction.
However, during training the error never decreases. I’m wondering if the error may come from inplace value assignment for fc3, fc4? My current setup is attached below.
class Surface_NN(torch.nn.Module):
  def __init__(self):
    super(Surface_NN, self).__init__()
    self.fc1 = nn.Linear(3, 25)
    self.fc2 = nn.Linear(25, 303)
    
    self.fc3 = nn.Linear(2, 50) # 3 * 50 = 150 params
    self.fc4 = nn.Linear(50, 3) # 51 * 3 = 153 params, in total 303 params

  def forward(self, Image, Surface):
    hidden = self.fc1(Image)
    hidden = F.relu(hidden)
    hidden = self.fc2(hidden)

    # NOTE THIS SPECIFIC SETUP ONLY APPLY TO BATCH_SIZE = 1.
    weights = hidden[0]

    fc3weight = (weights[0:100]).reshape([1, 50,2])
    fc3bias = weights[100:150]
    fc4weight = (weights[150:300]).reshape([1, 3, 50])
    fc4bias = weights[300:304]

    self.state_dict()['fc3.weight'] = fc3weight
    self.state_dict()['fc3.bias'] = fc3bias
    self.state_dict()['fc4.weight'] = fc4weight
    self.state_dict()['fc4.bias'] = fc4bias
 
    middle = F.relu(self.fc3(Surface))
    result = self.fc4(middle)
    return result