I don’t see how you can get better performances.
The forward and backward are sequential.
And if you say you have 2 device and you want to overlap multiple forward backward. Then you can overlap them while keeping each fw/bw pair on the same device as well.

Which APIs would be involved in such a task? How complex could this task be?
Could you give insight with respect to the existing APIs?

The whole backward pass assumes that intermediary gradients match the intermediary results in the forward pass. Also if some buffers were saved, they would need to be moved from different device.
This will for sure not be accepted as a PR (unless you can show a use case where it brings massive speed improvement). And hacking it might still be quite hard.