I think this is the case.
Here 5 the output tensor will be initialized with the device (options) of the first tensor.
The device check is here 4 and dispatches to parallel_cat.
If different devices are found (or any other condition returns false), the copy_ should push the tensors to the first device in these lines of code 4.
Could this explain the issue using multiple GPUs?