Here, distributions are blocking because of scalar parameters, that are being wrapped in tensors (and copied to GPU blocking python) - this is avoidable with pre-created tensors (even better is to use simplified formulas for scale=1)
I’m not sure about backward() snippet, I think writing parameter_cpu.grad is blocking. But sync before starting backprop may also make sense…