No, I don’t think you can optimizer both models together using different frameworks.
While the inference use case might work, I’m not aware of any tool allowing the backward pass to cross between frameworks.