My approach is hacky as described and if it’s not working on your particular model, I would recommend to create a custom model by deriving from your base model and to explicitly replace the layers you would like to change.