Hi,
I am trying to optimize an MLP using multi-losses that are computed at different times for the same neural network. Basically, the problem is a combination of supervised and semi-supervised learning.
I wish to optimize the algorithm using SGD with labels for each training example, and then would like to also optimize the network using SGD with labels for a set of training examples (e.g. sum of predictions over a certain period of time will have a label)
dataset = {clip_label: frame1_label:frame1,frame2_label: frame2, frame3_label:frame3}
# unrolled loop
optimizer.zero_grad()
output = net(frame1)
loss = criterion(output, frame1_label)
loss.backward()
 optimizer.step()
summed_output += torch.sum(output)  # sum predicted output

optimizer.zero_grad()
output = net(frame2)
loss = criterion(output, frame2_label)
loss.backward()
 optimizer.step()
summed_output += torch.sum(output)  # sum predicted output

optimizer.zero_grad()
output = net(frame3)
loss = criterion(output, frame3_label)
loss.backward()
 optimizer.step()
summed_output += torch.sum(output)  # sum predicted output



# Optimize at the clip_level 
optimizer.zero_grad() 
loss = criterion(summed_output , clip_label)
loss.backward()
optimizer.step()

This is how the code should look like. Zeroing the gradients might be problematic for the clip_level optimization. When I run my code, I get this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


If I detach summed_output from the graph, then this error disappears, but I do not think the training process is correct.
Any idea how to accomplish this process correctly?
Thanks