Solved by jpeg729 in post #2 
              
              
                Looking at the source for Adam, it loops over all parameter sets in all parameter groups, and does its calculations separately for each parameter set. 
    for group in self.param_groups:
        for p in group['params']:
            ...
            state = self.state[p]

            # State initial…