You should be able to use different architectures as long as the torch.cat operation is able to concatenate the output tensors from both models. I don’t know how your new model would be interpreted as it’s getting features from one model and predictions from another, but technically it should work.
Are you seeing any issues using your code?