Hello Alan, Somehow the Norm (Infinity) is always returning 1. and giving in-correct value. My code is as pointed out earlier this.
def l2_regu(mdl):
        l2_reg = None
        for W in mdl.parameters():
                if W.ndimension() < 2:
                        continue
                else:
                        w_tmp = W
                        if l2_reg is None:
                                l2_reg = (torch.max(torch.abs(w_tmp)))**2
                        else:   
                                l2_reg = l2_reg + (torch.max(torch.abs(w_tmp)))**2
      
        return l2_reg

Even though I kept my batch size as 4(very small), within an Epoch, I saw through nvidia-smi command the memory usage went from 100 MB to 12GB, throwing the CUDA MEMORY ERROR. If I simply replace that (torch.max(torch.abs(w_tmp)))**2 with l2 norm every thing works perfectly. I understand that there are 3 operations here as contrast to 1 operation in l2 norm, But I am getting memory error within an Epoch that too  for a very small batch size.