Yes that makes sense, and can be done indeed (this is actually a nice research topic).
Note that BigScience 2 is investigating this, they call it “prefix-lm”: Models - Hugging Face 3
I assume that bidirectional attention is applied to the “prefix” (i.e. context), and unidirectional on the generation part.