You could store the positions of all the non-zeros:
weights.nonzero()

(You can then use scatter to fluff the model back up again, after loading (edit: you might need to .view(-1) the tensor before running .nonzero(), and then reshape it after loading/scattering)).