Solved by markdaoust in post #3 
              
              
                My first guess is that your “loss function” should be an an instance of a class that has a build-in circular-memory buffer implemented in a tf.Variable. 
Make the buffer large enough that you always have the record you need to go back to look at. 
If you’re using keras, you’ll need to train_step so …