Looks like this is a batch size problem. When setting batch size to 1, this problem occurs.
Is there a way to make predictions indifferent of batch size? or should it always be the same as training value?