i pinpointed the problem but still not sure how to solve it.
also not sure if it’s possible to reproduce without the data.
still, i have a 2d tensor size: (96,96) with all elements 0’s except for 15 containing 1’s.
let’s call this tensor mask. so:
>>> import torch
import torchvision.transforms.functional as TF
deg = 20
mask.nonzero(as_tuple=False)
Out[44]: 
tensor([[27, 38],
        [27, 52],
        [28, 14],
        [29, 80],
        [36, 28],
        [36, 65],
        [37, 20],
        [37, 36],
        [37, 58],
        [37, 73],
        [59, 47],
        [71, 29],
        [71, 46],
        [71, 64],
        [82, 46]])
>>>mask = TF.to_pil_image(mask)
mask = TF.rotate(mask, deg)
mask = TF.to_tensor(mask).squeeze()
mask.nonzero(as_tuple=False)
Out[46]: 
tensor([[19, 72],
        [27, 45],
        [29, 68],
        [31, 60],
        [34, 54],
        [41,  9],
        [42, 33],
        [43, 25],
        [47, 18],
        [58, 51],
        [64, 71],
        [70, 54],
        [76, 38],
        [80, 58]])

if you count, notice that there were 15 non zero elements prior to transform and 14 after transform.
this is random and sometimes leads to less elements sometime to more elements, sometimes exactly to the desired 15 elements. of course the randomness might be caused by the random generator of degrees i didn’t mention in this comment