What is the reason for having the non_blocking flag in various functions that transfer tensors between devices?    And why is it set to False by default?
It is my understanding that all GPU operations are completed asynchronously by default (and, in fact, this is vital for achieving a good performance).   So why does pytorch make an exception for transfers?
The only reasons for not using non_blocking that are mentioned in the docs are benchmarking and CUDA streams.  However these are pretty niche use-cases, and one should probably use an explicit synchronization for those anyway.
I’ve been spraying my code with non_blocking=True on assumption that this will never hurt and might occasionally speed things up.  Am I wrong about that?