But have you solved the problem for the forward?
I have not looked into this in detail, but I think I saw discussions about using cuda streams matching the forward for the backward or so, which might help.
If your ensemble is structurally similar, it might also be worth trying to vectorize across models as an alternative. Or you could increase the batch size and do the ensemble calculations in sequence as a generic solution.
Best regards
Thomas