Solved by Ditlev_Jorgensen in post #4 
              
              
                I’ve found another guy who does exactly what you want to do (I think). And he also have to re-initialize the optimizer every time. Remember to resize bias as well not only weights. 
He is resizing weights during training: resizing weights during training 
However, as seen in the comments, you cannot…