Depending on the relative computational cost of the models, it maybe difficult to parallelize them across multiple GPUs “simultaneously” and synchronize the models across each batch. Since this is for pretrained models, can you do the predictions in a more “offline” way, where the first model processes all the data followed by the second model with the predictions being aggregated after both models are done?