Hi Kaushal,
I am also not too familiar with transformers library. It’s isn’t not achievable. But you might need little bit of effort.
I suggest you can try it inside you model layer itself. Decoding can be untouched. Just make a condition in your model, that at inference pass the extra embedding for concatenation. It’s fairly straightforward in PyTorch. But not so in TF2 , if you want to serialize.
I suggest try these in transformers PyTorch implementation. From my personal experience, their TF implementation is no where close to perfect ( which is no one’s fault ), TF requires effort.