Thank you for your response. I should state the problem more clearly. I’m using the same trained model for inference on two different machines and seeing results differ like this. I simplified the problem down to an unintialized network to see where the error was coming from.
To predict, this network is called multiple times parts of a time series and the likelihood for labels is continuously updated. So this slight difference here eventually diverges enough on until the prediction differ.
Since this seems to be a computation issue do you have any advice for keeping things more stable? I’ve tried change the data precision and manually rounding variables. I’m also thinking about ways to make my prediction step less accumulative? Do you have any experience predicting long time series and keeping the predictions real time (eg. outputting results frequently while keeping the RNN state)?