Hi Iram!
Let’s start with the first three lines of the code you posted:



 ArshadIram:

maxEpochs = 100
z = nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to(device)
opt = torch.optim.Adam([z]).to(device)



Let me assume that device refers to a gpu.
Your second line of code creates a tensor on the cpu and then
“moves” it to the gpu by creating a gpu copy of the original cpu
tensor.  Although the original cpu tensor is a “leaf” tensor, the
copy-to-gpu operation counts as a “computation” so that the
gpu tensor is not a leaf tensor, hence the error you report:


I am getting the error
ValueError(“can’t optimize a non-leaf Tensor”)


Consider:
>>> import torch
>>> torch.__version__
'1.9.0'
>>> x = torch.nn.Parameter(torch.rand(64,100,1,1), requires_grad=True)
>>> x.is_cuda
False
>>> x.is_leaf
True
>>> y = torch.nn.Parameter(torch.rand(64,100,1,1, device = 'cuda'), requires_grad=True)
>>> y.is_cuda
True
>>> y.is_leaf
True
>>> z = torch.nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to ('cuda')
>>> z.is_cuda
True
>>> z.is_leaf
False

As an aside, your third line of code, as posted, is fully bogus, and
will throw an error, even if you try to construct your Adam optimizer
with a leaf tensor.  (In general, a pytorch Optimizer doesn’t have a
.to() method.)
>>> opt = torch.optim.Adam([x]).to ('cuda')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Adam' object has no attribute 'to'

Construct the Parameter you wish to optimize directly on the gpu, as
is done with y in the example I gave above, and then use it to construct
the Optimizer (but don’t attempt to “move” the Optimizer):
>>> opt = torch.optim.Adam([y])
>>> opt.param_groups[0]['params'][0].shape
torch.Size([64, 100, 1, 1])
>>> opt.param_groups[0]['params'][0].is_cuda
True
>>> opt.param_groups[0]['params'][0].is_leaf
True

Best.
K. Frank