The following issue indicates that cat is slow on CPU: https://github.com/pytorch/pytorch/issues/18634 88
I found that a custom cat function is slightly faster on CPU:
def customCat(allTensors):
    totalSize = sum([t.shape[0] for t in allTensors]);
    newTensor = torch.zeros(totalSize, allTensors[0].shape[1], allTensors[0].shape[2]);
    counter = 0;
    for t in allTensors:
        newTensor[counter:counter+t.shape[0]] = t;
        counter += t.shape[0];
    return newTensor;

It definitely isn’t a generic replacement for cat, and I haven’t tested this code.
You also have the option of doing concatenations in the DataLoader via a user-defined collate_fn. DataLoader will be able to parallelize the cat operations across multiple batches. It may hide any latency you see.