.wait_stream() is actually not synchronous; it just means that future operations submitted to a stream will not execute until after another stream’s currently queued operations have completed.
https://pytorch.org/docs/stable/cuda.html#torch.cuda.Stream.wait_stream 2
It’s just meant to prevent the parallel streams from running until W has been initialized.
If I take out the wait_stream() calls, it still takes longer anyway:
result: 9894706.0
time: 0:00:04.524583
memory: 364826624