I actually got to observe something interesting(which I didn’t understand).
As pointed out earlier, memory  error was only seen for the case where I set requires_grad = true for Ident matrix.  But If I simply use other norm such as l2 using torch.norm(w,2) instead of torch.max(torch.abs(w)), even through the require_grad is True, I don’t see the memory error:
I Mean this code , doesn’t throw the memory error:
def reg(mdl):
        l2_reg = None
        for W in mdl.parameters():
                if W.ndimension() < 2:
                        continue
                else:
                        wt = torch.transpose(W,0,1)
                        m  = torch.matmul(wt,W)
                        **ident = Variable(torch.eye(cols,cols), requires_grad=True)**
                        ident = ident.cuda()
                        w_tmp = (m - ident)
                        if l2_reg is None:
                                l2_reg = (torch.norm(w_tmp,2))**2
                        else:
                               l2_reg = l2_reg + torch.norm(w_tmp,2)**2

From here there seems to be some difference between the torch.norm and torch.abs(torch.max()), which we seems to be missing.