Seems viable to me. That’ll be following the idea of “Learn to work with the decoder first, then we can train you both jointly”. I don’t think there’s a wrong way to approach it; Maybe consider trying out the different methods and seeing if it even makes a difference