Hi Cai!



 Chen-Cai-OSU:

What I want is to fix the lower part to be 0 all the time. I think I have to unregister those numbers but I didn’t find a clean way to do so.


An entire tensor is a single Parameter.  You can’t unregister just some
elements of a tensor.
Your best bet will be to “freeze” the lower triangle at zero by zeroing out
that part of the tensor’s gradient.  Here’s a post from @ptrblck that shows
how to do this:

  
    
    
    Creating a Triangular weight matrix with a hidden Layer instead of a fully connected one 
  
  
    You could initialize the weight matrices in your desired shape, e.g. by using torch.triu, and then zeroing out the gradients at the same locations. 
Here is a small code snippet for this use case, which could be a good starter: 
# Init
def triu_init(m):
    if isinstance(m, nn.Linear):
        with torch.no_grad():
            m.weight.copy_(torch.triu(m.weight))

# Zero out gradients
def get_zero_grad_hook(mask):
    def hook(grad):
        return grad * mask
    return hook

# Setup
model = nn…
  


Best.
K. Frank