You know it actually worked fine. I needed to add clone() to avoid any in_place operations… (not actually sure why this worked)
x = F.leaky_relu(self.fc1(x)) # [32, 225]

for i in range(len(x[:,])):
            s = x[i, :]
            s = s.clone() * self.phi().clone()
            s = get_top_k(s, 10, mask_type="binary")
            self.phi.update(s, 0.1618)
            x[i, :] = s

and
loss.backward(retain_graph = True)


It sure was slow though!
I did try stack as you suggested (with unbind as well), but it didn’t really seem to give me any boost and was a little more verbose…
        s_list = []

        x = F.leaky_relu(self.fc1(x)) # [32, 225]

        samples = torch.unbind(x)
        for s in samples:
            s = s.clone() * self.phi().clone()
            s = get_top_k(s , 10, mask_type="binary")
            self.phi.update(s, 0.1618)
            s_list.append(s)
        x = torch.stack(s_list).squeeze()

Really slow, but adding clone() helped backward() make sense of it all, though I am not sure why.