Solved by ptrblck in post #2 
              
              
                I think your use case should work if you wrap the inplace manipulations of param into a with torch.no_grad() guard as seen here: 
import torch
import torch.nn as nn
import torch.nn.functional as F

lower_bound, upper_bound = 0., 1.
param = nn.Parameter(torch.randn(10, 10) * 10.)
optimizer = torch.op…