Body: <p>I have a model for binary classification. The target variable has the different number of labels (instances) in each sample. For example, a batch of size 2 with 2 and 3 instances and correspondingly with 2 and 3 labels (0 and 1):</p>
  <pre><code>y_true = np.array([[0., 1., -1., -1.],
                     [1., 0., 1., -1.]])
  </code></pre>
  <p>The maximal number of labels (instances) in a sample is equal to 4. -1 values are used as a mask. I created a function (in TensorFlow) that masks all -1 values and calculates the loss for each unmasked value and then the average of all losses:</p>
  <pre><code>def my_loss_fn(y_true, y_pred):
      mask = tf.cast(tf.math.not_equal(y_true, tf.constant(-1.)), tf.float32)
      y_true, y_pred = tf.expand_dims(y_true, axis=-1), tf.expand_dims(y_pred, axis=-1)
      bce = tf.keras.losses.BinaryCrossentropy(reduction='none')
      return tf.reduce_sum(tf.cast(bce(y_true, y_pred), tf.float32) * mask) / tf.reduce_sum(mask)
  </code></pre>
  <p>Is it a correct way mathematically? Should I add some weights depending on number of labels in a sample? Or calculate loss per sample (per row in my example) and then get the average? Should I mask unused labels at all? It looks like my model doesn't learn to classify correctly and predicts 1 for all labels.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-10T18:06:27.983Z
Comments.Text: Correct for what model? The ordinary categorical cross-entropy loss arises from a specific probability model of the data. The categorical cross-entropy loss is the "correct" loss for that probability model, but it's "incorrect" if the data are assumed to arise from a different probability model. This reads a bit like an 
  [XY Problem](http://xyproblem.info/). What problem are you trying to solve, and how does having a variable number of classes for each instance fit into solving it?
Comments.UserDisplayName: Sycorax
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-10T18:06:34.360Z
Comments.Text: Here's one possible solution. For a single instance, the categorical cross-entropy loss for $K$ classes is
  
  $$
  L=-\sum_{k=1}^K y_k \log p_k=0 + \dots + 0 - \log p_t + 0 + \dots + 0
  $$
  where $y_k$ is the one-hot class label and $p_k$ is the estimated probability that the instance belongs to class $k$. Because $y_k$ is one-hot, there's only one nonzero term in the sum. The only part of this that involves the number of classes $K$ is the fact that there are $K-1$ zero entries in the sum. If this is the model you want, then no weighting or other refinement is needed!
Comments.UserDisplayName: Sycorax
Tags: <deep-learning><loss><binary-classification><binary-crossentropy>
Title: Categorical loss function for variable number of labels
UserDisplayName: Mykola Zotko
ThreadText: # Categorical loss function for variable number of labels
  
  (Asked by: username_0 on 2022-08-10)
  
  I have a model for binary classification. The target variable has the different number of labels (instances) in each sample. For example, a batch of size 2 with 2 and 3 instances and correspondingly with 2 and 3 labels (0 and 1):
  y_true = np.array([[0., 1., -1., -1.],
                     [1., 0., 1., -1.]])
  
  The maximal number of labels (instances) in a sample is equal to 4. -1 values are used as a mask. I created a function (in TensorFlow) that masks all -1 values and calculates the loss for each unmasked value and then the average of all losses:
  def my_loss_fn(y_true, y_pred):
      mask = tf.cast(tf.math.not_equal(y_true, tf.constant(-1.)), tf.float32)
      y_true, y_pred = tf.expand_dims(y_true, axis=-1), tf.expand_dims(y_pred, axis=-1)
      bce = tf.keras.losses.BinaryCrossentropy(reduction='none')
      return tf.reduce_sum(tf.cast(bce(y_true, y_pred), tf.float32) * mask) / tf.reduce_sum(mask)
  
  Is it a correct way mathematically? Should I add some weights depending on number of labels in a sample? Or calculate loss per sample (per row in my example) and then get the average? Should I mask unused labels at all? It looks like my model doesn't learn to classify correctly and predicts 1 for all labels.
  
  **Tags:** deep-learning, loss, binary-classification, binary-crossentropy
  > **Comments:**
  > **username_1**: Correct for what model? The ordinary categorical cross-entropy loss arises from a specific probability model of the data. The categorical cross-entropy loss is the "correct" loss for that probability model, but it's "incorrect" if the data are assumed to arise from a different probability model. This reads a bit like an  [XY Problem](http://xyproblem.info/). What problem are you trying to solve, and how does having a variable number of classes for each instance fit into solving it?  
  > **username_1**: Here's one possible solution. For a single instance, the categorical cross-entropy loss for $K$ classes is  $$ L=-\sum_{k=1}^K y_k \log p_k=0 + \dots + 0 - \log p_t + 0 + \dots + 0 $$ where $y_k$ is the one-hot class label and $p_k$ is the estimated probability that the instance belongs to class $k$. Because $y_k$ is one-hot, there's only one nonzero term in the sum. The only part of this that involves the number of classes $K$ is the fact that there are $K-1$ zero entries in the sum. If this is the model you want, then no weighting or other refinement is needed!  
  
  ---
  
  ## 0 answers
  
  