If these layers were previously updated, i.e. they got a valid gradient in previous iterations, are now frozen via .requires_grad=True, optimizers with running stats (such as Adam) can still update these parameters.
To avoid this, you could try to set their .grad attributes to None, which should skip the updates.