TomerF:

Is it acceptable to set Layer2.requires_grad_(False) while I compute the forward pass for LossPartial and then to set Layer2.requires_grad_(True) for computing the forward pass for LossAll?


Yes it is  That should work as you expect.