you are absolutely right. The reason why predictions differ during eval phase is due to the non-determinism. I confirmed this by evaluating the same inputs several time, and the results vary from run to run.