Alright!  So the model.eval() made sense for the loss changing due to the Dropout ( since it bypasses that model in eval), but I initially had no Dropout and the model.eval mode still didn’t work for me.
Turns out I was randomly initializing the “initial state” of my LSTM  Hence the randomness in the losses for the same data over and over again. I fixed that by initializing with zeros and added a Dropout layer ( all just to learn how it works) and then used the eval method and sure enough I get this!
The Code ( from the POS example in Pytorch Docs):

model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))
loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)
with torch.no_grad():
inputs = prepare_sequence(training_data[0][0], word_to_ix)
tag_scores = model(inputs)
def _train(epoch):
epoch_loss = 0
for sentence, tags in training_data:
    model.zero_grad()
    model.hidden = model.init_hidden()

    sentence_in = prepare_sequence(sentence, word_to_ix)
    targets = prepare_sequence(tags, tag_to_ix)
    tag_scores = model(sentence_in)
    loss = loss_function(tag_scores, targets)
    epoch_loss+=loss
    
    loss.backward()
    optimizer.step()
    
print("EPOCH ",epoch,":",epoch_loss.item())
return

def _evaluate(dataset):
epoch_loss = 0
for sentence, tags in dataset:
    model.zero_grad()
    model.hidden = model.init_hidden()

    sentence_in = prepare_sequence(sentence, word_to_ix)
    targets = prepare_sequence(tags, tag_to_ix)
    tag_scores = model(sentence_in)
    loss = loss_function(tag_scores, targets)
    epoch_loss+=loss
    # REMOVED THE BACKPROP LOSS CODE FOUND IN _TRAIN METHOD

print("EPOCH RERUN ON TRAINING :",epoch_loss.item())

for epoch in range(300):  # again, normally you would NOT do 300 epochs, it is toy data
model.train()
_train(epoch)
model.eval()
_evaluate(training_data)
_evaluate(training_data)
print("\n")

And I get this :

EPOCH  0 : 2.2410192489624023
EPOCH RERUN ON TRAINING : 2.171501636505127
EPOCH RERUN ON TRAINING : 2.171501636505127
EPOCH  1 : 2.1357598304748535
EPOCH RERUN ON TRAINING : 2.1666126251220703
EPOCH RERUN ON TRAINING : 2.1666126251220703
EPOCH  2 : 2.1036343574523926
EPOCH RERUN ON TRAINING : 2.1608502864837646
EPOCH RERUN ON TRAINING : 2.1608502864837646
EPOCH  3 : 2.153252363204956
EPOCH RERUN ON TRAINING : 2.1550378799438477
EPOCH RERUN ON TRAINING : 2.1550378799438477
EPOCH  4 : 2.242952823638916
EPOCH RERUN ON TRAINING : 2.1500635147094727
EPOCH RERUN ON TRAINING : 2.1500635147094727
EPOCH  5 : 2.2232353687286377
EPOCH RERUN ON TRAINING : 2.1464695930480957
EPOCH RERUN ON TRAINING : 2.1464695930480957
EPOCH  6 : 2.2370853424072266
EPOCH RERUN ON TRAINING : 2.1424150466918945
EPOCH RERUN ON TRAINING : 2.1424150466918945
EPOCH  7 : 2.152524948120117
EPOCH RERUN ON TRAINING : 2.1402010917663574
EPOCH RERUN ON TRAINING : 2.1402010917663574
EPOCH  8 : 2.0829806327819824
EPOCH RERUN ON TRAINING : 2.137516736984253
EPOCH RERUN ON TRAINING : 2.137516736984253
EPOCH  9 : 2.1681880950927734
EPOCH RERUN ON TRAINING : 2.1328389644622803
EPOCH RERUN ON TRAINING : 2.1328389644622803

Which is great! Since After EPOCH loss, the model backprops so we know that the RERUN loss won’t be the same as the EPOCH loss. We also see that the RERUN loss is deterministic in eval model!
My last question is that : what changes in-between epochs? 
Why isn’t the current EPOCH loss equal to the RERUN loss from the previous epoch ?