As far as I know, if one permutes a matrix, that is used as a fully-connected layer, the model should learn identically, compared to any other permutation?  ie if I do:
self.w = nn.Parameter(torch.Tensor(batch_size, neurons))
self.w.data.uniform(-stdv, stdv)
self.w.data = permute(self.w.data.view(-1)).view(batch_size,neurons)
...
x = x @ self.w

… the model will converge exactly the same, no matter what is the permutation inside permute?
(edited, to correct a bit…)