albanD:

So you do any of these (and mix) because the different part of your model are independent 


Just to be sure I got your point: gradients are independent for each row in my input dataset, and even if I mix all of them in the loss, autograd is still aware of it and it treats each row/time series in the exact same way (in terms of updating the parameters’ values when I do optim.step()) as if I were to compute the backward pass for each one of them independently, right?



 albanD:

Note that in general, it is better to sum the losses and do a single backward (speed wise).


Are you suggesting that I change my current loss, from:
def loss(self,p):
        out = kmodel(p,self.IF,self.device) - self.y
        out = torch.sum((out)**2, dim=1)
        return torch.mean(out)

to
def loss(self,p):
        out = kmodel(p,self.IF,self.device) - self.y
        out = torch.sum((out)**2, dim=1)
        return torch.sum(out)

or
def loss(self,p):
        out = kmodel(p,self.IF,self.device) - self.y
        return torch.sum((out)**2)

Will any of these options make any difference in terms of speed/memory usage?
I am especially interested in this because despite this toy example, in a real case for me Nv should be in the order of millions …