I should also add that I’m asking this because when I don’t load the state dict for these layers prior to training, and instead just train the entire model (all layers) at once, I’m getting decent performance, although taking longer than if it were fewer layers.
When I load the state dict from the saved model like this, I don’t see any indication that anything is adjusting over time. It looks like nothing is being updated.