For the layers/modules that you do not want to train, you can set their requires_grad=False which would also save you GPU memory. To make your modules non-trainable you can use {model}.{submodule_name}.requires_grad_(False).