I don’t think I understand the proposed model architectures completely, but would claim your idea is feasible since you can combine models as you wish. As long as you don’t detach the computation graph (e.g. via rewrapping a tensor, using another library such as numpy etc.) it should work.