If you are using torch.cat in a loop, you should instead append all tensors to a list and create the tensor out side of the loop. Also, you could try to preallocate the output tensor and fill it. If this operation would break your training, you should get an error and would have to fall back to the first approach.