The error from adding already frozen parameters to the optimizer is an explicit design choice to avoid silly mistakes 174 (one could mistakenly add the frozen one and assume that the parameter is being trained). But manually freezing parameters after declaring the optimizer works as intended without re-defining the optimizer. (my own replay above in this thread)