I recently profiled my model code and one surprising thing that came up was how slow checking if a tensor is all 0’s is.  More specifically,
my_tensor.sum().data[0] == 0
takes upwards of 0.01 seconds per execution which can add up. I’m guessing this is because ‘my_tensor’ is a variable on GPU and must be converted to CPU before doing the boolean check. My question is whether there is a faster way to check whether ‘my_tensor’ is all 0’s (i.e. turn into a Boolean True/False) statement. Assume ‘my_tensor’ needs to be  GPU variable.
I’m thinking of doing,
len(torch.nonzero(my_tensor)) < 1
Theoretically this should work but it seems a bit hacky.