Is the result always a square or always continuous between rows columns?
nonzeros = torch.nonzero(tensor != DUMMY)
c_min, c_max, r_min, r_max = nonzeros[:, 0].min(), nonzeros[:, 0].max(), nonzeros[:, 1].min(), nonzeros[:, 1].max()
tensor = tensor[c_min:c_max+1, r_min:r_max+1]