You could use two optimizers, one for the CPU parameters and the other one for the GPU parameters.
The internal states of Adam should be parameter-dependent and thus wouldn’t create a bad interaction, would they?