Hi,
I guess you the problem is your update_weight_manually function: It is always updated in a differentiable manner, meaning that it will keep track of all it’s history.
You want to make sure that at each forward, you work with a self.weight that has no history.
You want to make sure this does not happen with:
    def update_weight_manually(self):
        self.weight = self.weight.detach()
        self.weight[self.adjacency==1] = self.weight_learnable

Also your current implementation might keep non-zero values from one forward to the next if the adjacency changes. If the adjacency can change, you might want to replace the detach by self.weight=torch.zeros([self.out_features,self.in_features]).