Solved by justusschock in post #4 
              
              
                Would intermediate backprops work? 
For example 
optimizer.zero_grad() 
x1 = self.cnn(x1)
... # calculate your loss
loss.backward()
optimizer.step()
optimizer.zero_grad() 
x2 = self.cnn(x2)
... # calculate your loss
loss.backward()
optimizer.step()
optimizer.zero_grad() 
x3 = self.cnn(x3)
... # calc…