Instead of freezing the layers as per documents (using require_grad = False), would it equivalent to pass just the trainable parameters and not specify the “frozen” parameters.
I imagine this case should would work as expected:
Frozen Layer (not given to optimizer) -> Trainable Layer -> OUTPUT
But I am unsure if this would be OK:
Trainable Layer -> Frozen Layer (not given to optimizer) -> OUTPUT
I am unsure because gradients will accumulate still (?) in the frozen layer during the loss backpropagation, and would this effect the ability of the optimizer to properly optimize the trainable layer?