Say you want to optimize parameters of your model where your parameters are a tensor n x D.
var1 = torch.tensor(..., requires_grad = True) # Tensor of shape n x D
optimizer = torch.optim.Adam([var1], lr = 1e-5)
for ite in range(ite_max):
     optimizer.zero_grad()
     loss = loss_function(var1)
    optimizer.step()

I want to add an orthogonal constraint on my parameter var1, that is,
var1[:,i].T * var1[:,j] = kronecker_ij

where kronecker_ij = 1 if i = j, 0 otherwise.
How should I do it?