I feel probably run everything on CPU would be better, running one op in CUDA sounds a bit weird…
not sure about your goal though, if you find running torch.cat on CUDA is faster and you need to squeeze every bit of perf then maybe that is OK as well