After some more debugging, it turned out that my model was close to the point of going OOM since the beginning but only when input with large number of batch size arrived it actually crashed. I was able to delay this event of crash by doing del logits in the end which apparently gave more room for the larger batch size but nevertheless it crashed after many epochs. The reason why model is not able to fit in the gpu seems a bit bizarre and I have opened separate discussion thread 260 regarding that.
Many many thanks for support