No problem @valhalla . Appreciate your response.
I have implemented this feature locally.
Off topic, would like to know what is your thoughts on this.

  
    
    
    Why Tensorflow Models are way slower than Pytorch models, for autoregressive modeling? 🤗Transformers
  
  
    Hi, 
I was experimenting with many models include GPT2, T5 etc. But it seems like Tensorflow models are too slow for same type of generation comparing to Tensorflow, whether it is greedy, beam etc . 
Any specific reasons for this? 
Thanks