My first guess is that your “loss function” should be an an instance of a class that has a build-in circular-memory buffer implemented in a tf.Variable.
Make the buffer large enough that you always have the record you need to go back to look at.
If you’re using keras, you’ll need to train_step so you can thread the bars_in_x feature through to the loss function.

  
      

      TensorFlow
  

  
    

Customize what happens in Model.fit  |  TensorFlow Core