This use case is not supported by the Trainer API, it would require custom scripts (one per GPU) to work, and even then, I’m not sure you will see any speed gain by training on two different GPUs that are not of the same type.