I dont think you can change the size of your model during training. The optimizer is set to update a specific set of parameters. When you add new parameters, the optimizer dont know these exists, and thus needs to be reinitialized with the newly added parameters in order to work again.