You can add new units to your optimizer and keep same hyperparameters without recreating it by using add_param_group like this:
optimizer.add_param_group({'params': [p for p in new_units.parameters()]})

where new_units in this case is a list of modules containing parameters.
Look here 27 for the official docs.
Hope this helps.