Thank you @Samarth_Mishra for trying to help. After going through it line-by-line, I have identified the problem. And as weird as it sounds, it has nothing to do with the model. It was this line.
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', additional_special_tokens=['_eos', '_go'])

For some strange reason, the BertTokenizer randomly assigns different index to _eos and _go each time the code is run. I suspect they might use their own custom random function. That was probably why the seed we set does not affect this.
Simply change the line to this, fix my problem.
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(['_eos', '_go'])