Looking at the source for Adam 5, it loops over all parameter sets in all parameter groups, and does its calculations separately for each parameter set.
    for group in self.param_groups:
        for p in group['params']:
            ...
            state = self.state[p]

            # State initialization
            if len(state) == 0:
                ...

            # do calculations
            ...

So when you add a new parameter group to the optimiser using optimizer.add_param_group(({'name':optional, 'params':new_module.parameters() })) 10 the old parameters will keep their existing optimiser state, and the new parameters optimiser state will be initialised correctly.
Other optimisers should work similarly.