Torch.cat is just putting one tensor at the end of another so this is a very small calculation (if you can call it that) so this will run slower on a GPU compared to CPU due to the small data size and small action that is needed. So it will take time for the code to send this tensor to a GPU as it is overhead for the GPU to start and read in everything. Similar to CPU where it has to read in things,  but a CPU does not have the starting overhead.
So by the time the GPU starts up, the CPU would have finished the calculation