title: Doing Multi-Label classification with BERT
body: <p>I want to use <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">BERT model</a> to do multi-label classification with Tensorflow. </p>
  
  <p>To do so, I want to adapt the example <code>run_classifier.py</code> from <a href="https://github.com/google-research/bert" rel="noreferrer">BERT github repository</a>, which is an example on how to use BERT to do simple classification, using the <a href="https://github.com/google-research/bert#pre-trained-models" rel="noreferrer">pre-trained weights given by Google Research</a>. (For example with <code>BERT-Base, Cased</code>)</p>
  
  <p>I have <code>X</code> different labels which have value of either 0 or 1, so I want to add to the original BERT model a new Dense layer of size <code>X</code> and using the <code>sigmoid_cross_entropy_with_logits</code> activation function.</p>
  
  <p>So, for the theorical part I think I am OK.</p>
  
  <p>The problem is that I don't know how I can append a new output layer and retrain only this new layer with my dataset, using the existing <code>BertModel</code> class.</p>
  
  <p>Here is the original <code>create_model()</code> function from <code>run_classifier.py</code> where I guess I have to do my modifications. But I am a bit lost on what to do.</p>
  
  <pre><code>def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
                   labels, num_labels, use_one_hot_embeddings):
    """Creates a classification model."""
    model = modeling.BertModel(
        config=bert_config,
        is_training=is_training,
        input_ids=input_ids,
        input_mask=input_mask,
        token_type_ids=segment_ids,
        use_one_hot_embeddings=use_one_hot_embeddings)
  
    output_layer = model.get_pooled_output()
  
    hidden_size = output_layer.shape[-1].value
  
    output_weights = tf.get_variable(
        "output_weights", [num_labels, hidden_size],
        initializer=tf.truncated_normal_initializer(stddev=0.02))
  
    output_bias = tf.get_variable(
        "output_bias", [num_labels], initializer=tf.zeros_initializer())
  
    with tf.variable_scope("loss"):
      if is_training:
        # I.e., 0.1 dropout
        output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)
  
      logits = tf.matmul(output_layer, output_weights, transpose_b=True)
      logits = tf.nn.bias_add(logits, output_bias)
      probabilities = tf.nn.softmax(logits, axis=-1)
      log_probs = tf.nn.log_softmax(logits, axis=-1)
  
      one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
  
      per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
      loss = tf.reduce_mean(per_example_loss)
  
      return (loss, per_example_loss, logits, probabilities)
  </code></pre>
  
  <p>And here is the same function, with some of my modifications, but where there is things missing (and wrong things too? )</p>
  
  <pre><code>def create_model(bert_config, is_training, input_ids, input_mask, segment_ids, labels, num_labels):
    """Creates a classification model."""
    model = modeling.BertModel(
        config=bert_config,
        is_training=is_training,
        input_ids=input_ids,
        input_mask=input_mask,
        token_type_ids=segment_ids)
  
    output_layer = model.get_pooled_output()
  
    hidden_size = output_layer.shape[-1].value
  
    output_weights = tf.get_variable("output_weights", [num_labels, hidden_size],initializer=tf.truncated_normal_initializer(stddev=0.02))
  
    output_bias = tf.get_variable("output_bias", [num_labels], initializer=tf.zeros_initializer())
  
    with tf.variable_scope("loss"):
      if is_training:
        # I.e., 0.1 dropout
        output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)
  
      logits = tf.matmul(output_layer, output_weights, transpose_b=True)
      logits = tf.nn.bias_add(logits, output_bias)
      probabilities = tf.nn.softmax(logits, axis=-1)
      log_probs = tf.nn.log_softmax(logits, axis=-1)
  
      per_example_loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits)
  
      loss = tf.reduce_mean(per_example_loss)
  
      return (loss, per_example_loss, logits, probabilities)
  </code></pre>
  
  <p>The other things I have adapted in the code and for which I had no problem : </p>
  
  <ul>
  <li>DataProcessor to load and parse my custom dataset</li>
  <li>Changing the type of labels variable from numerical values to arrays everywhere it is used</li>
  </ul>
  
  <p>So, if anyone knows what I should do to resolve my problem, or even point out some obvious mistake I may have done, I would be glad to hear it.</p>
  
  <p>Notes : </p>
  
  <ul>
  <li>I found <a href="https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d" rel="noreferrer">this article</a> that correspond pretty well to what I am trying to do, but it use PyTorch, and I can not translate it into Tensorflow.</li>
  </ul>