Hooo one more proof for https://github.com/pytorch/pytorch/issues/30987 35 that .data  is the source of all evil !
The short answer is: do not use .data 
The longer answer is:
m.weight= nn.Parameter(torch.cat((m.weight[:filter[1]], m.weight[filter[1]+1:])))

And make sure to give these new weights to your optimizer again after changing them !