title: Keras custom loss function: ValueError in a tf.function-decorated
body: <p>I'm trying to implement in TF 2.2 the loss function from <a href="https://arxiv.org/abs/1911.08623" rel="nofollow noreferrer">this paper</a> (an existing version in TensorFlow 1.10.1, made by the author of the paper can be found <a href="https://github.com/GuansongPang/deviation-network" rel="nofollow noreferrer">here</a>).</p>
  <p>However, the theoretical details of the loss function are not relevant to my problem.</p>
  <p>My loss function:</p>
  <pre><code>def z_score_based_deviation_loss(y_true, y_pred):
      confidence_margin = 5.0     
      ref = K.variable(np.random.normal(loc=0., scale=1.0, size=5000), dtype='float32')
      dev = (y_pred - K.mean(ref)) / K.std(ref)
      inlier_loss = K.abs(dev) 
      outlier_loss = K.abs(K.maximum(confidence_margin - dev, 0.))
      return K.mean((1 - y_true) * inlier_loss + y_true * outlier_loss)
  </code></pre>
  <p>Calling it by:</p>
  <pre><code>model.compile(optimizer='adam', loss=z_score_based_deviation_loss)
  </code></pre>
  <p>I got this error:</p>
  <pre><code>ValueError: tf.function-decorated function tried to create variables on non-first call.
  </code></pre>
  <p>I know there are other questions on this subject, but I haven't found any that are related to a custom loss function in Keras.
  I can't figure out how to adapt it.</p>
  <p>Also, from what I've read, the problem should exists when the function is tagged with <code>@tf.function</code> (which is recommended anyway to speed up the computation), but I didn't add it...</p>