correct. pretty much anything going to gpu, forward prop, or whatever, is in general asynchronous, unless something forces otherwise. Things that cause sync poitns:

calling ‘sync’ 

memory allocation
displaying values on the host
anything that causes values to be copied to the host