Hi Louis-Vincent!



 lv_Poellhuber:

When I apply the loss, would it backpropagate properly? I’m afraid it would only apply the loss to the LSTM and not the CNN, since they’re not directly connected.


I haven’t looked at your code, but yes, in general the kind of thing
you propose doing should work just fine.
Make sure you understand the basics of how pytorch uses the
require_grad property of a tensor.  Normally the weights, etc.,
of your model, and the tensors that depend on them, will have
requires_grad = True.  The input to your CNN would normally
have requires_grad = False.  However, the output of your CNN
will have requires_grad = True (because it depends on the
weights of the CNN).  The “connection” between the LSTM and
the CNN is provided by the fact that the output of your CNN – that
has requires_grad = True – is also the input to your LSTM.  This
is what permits pytorch’s autograd system to backpropagate your
final loss through the LSTM all the way back to the weights of your
upstream CNN.
Best.
K. Frank