Yes, the nonzero would need some time to be executed, but you are right that your profiling is wrong as it accumulates the model forward time into the nonzero operation, so it looks as if the nonzero op is more expensive than it is.
As mentioned before, this operation is synchronizing, so should be avoided for this reason (in case you want to remove all syncs, if possible).



 markcheung:

But I believe these indexing operations should be less effective than convolutions when the code is running on GPU.


I would claim it depends on the actual workflow, so you could profile it with synchronizations and compare the runtimes, if needed.