I’ve tested this and it doesn’t seem to error on my side. For example, if we modify the MNIST pytorch example 48, by changing lines 70-75 from
model = Net()
if args.cuda:
    model.cuda()

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

to
model = Net()

optimizer = optim.Adam(model.parameters(), lr=args.lr)

if args.cuda:
    model.cuda()

the code still runs fine (no error) and gives you expected results (e.g. 95% accuracy after 1 epoch). If you print the optimizer.state dict, it returns something like this in both cases (i.e. when model was put on GPU before and after creating optimizer):
defaultdict(<type 'dict'>, {Parameter containing:
-0.0099
-0.0659
-0.4953
-0.0524
-0.2689
-0.1911
-0.0981
-0.1887
 0.1477
-0.1847
 0.0446
-0.0893
-0.2479
-0.0239
 0.0504
-0.1310
-0.1203
-0.1727
-0.1563
-0.0666
[torch.cuda.FloatTensor of size 20 (GPU 0)]
: {'exp_avg_sq': 
1.00000e-04 *
  7.2218
  0.0056
  1.4282
...