title: Tensorflow - transfer learning implementation (semantic segmentation)
body: <p>I'm working on implementing a CNN architecture (FCN-8s model, with pretrained VGG16 model) for semantic segmentation on my own data (2 classes, therefore, a binary per-pixel classification)</p>
  
  <p>How I intend to go about this is:</p>
  
  <ol>
  <li>Load the pre-trained model with weights</li>
  <li>Add/remove additional higher layers to convert to FCN</li>
  <li>Freeze lower layers of the pre-trained model (to not update during the training phase)</li>
  <li>Train the network on specific dataset</li>
  </ol>
  
  <p>Assuming this is correct, how do I go about freezing the lower layers on my tensorflow model? (I'm looking for specific implementation details) I had a look at the Inception retraining on TensorFlow tutorial, but I'm not quite sure yet. </p>
  
  <p>This is the workflow I have in mind:</p>
  
  <ol>
  <li><p>Run my data through the existing pretrained model, and extract the feature outputs, <em>without</em> training it. (how?) </p></li>
  <li><p>Feed these feature outputs into another network containing the higher layers - and go about training it. </p></li>
  </ol>
  
  <p>Any suggestions would be helpful!</p>
  
  <p>Else, if I'm wrong, how should I be thinking of this?</p>
  
  <p>UPDATE:</p>
  
  <p>I took up <a href="https://stackoverflow.com/users/2876799/chasep255">chasep255</a>'s suggestion below, and tried to use <a href="https://www.tensorflow.org/versions/r0.9/api_docs/python/train.html#stop_gradient" rel="nofollow noreferrer">tf.stop_gradient</a> so as to "freeze" the lower layers in my model. Clearly, there is something wrong with my implementation. Possible alternatives/suggestions?</p>
  
  <p>The model is built based on the FCN (for semantic segmentation) paper. I extract <code>logits</code> from the model architecture, i.e., my features, that I initially feed directly into a <code>loss</code> function to minimize it with a softmax classifier. (per-pixel classification) <code>deconv_1</code> is my logits tensor, of shape <code>[batch, h, w, num_classes] = [1, 750, 750, 2]</code> Implementation:</p>
  
  <pre><code>logits = vgg_fcn.deconv_1
  
  stopper = tf.stop_gradient(logits, 'stop_gradients')
  
  loss = train_func.loss(stopper, labels_placeholder, 2)
  
  with tf.name_scope('Optimizer'):
      train_op = train_func.training(loss, FLAGS.learning_rate)
  
      with tf.name_scope('Accuracy'):
          eval_correct = train_func.accuracy_eval(logits, labels_placeholder)
          accuracy_summary = tf.scalar_summary('Accuracy', eval_correct)
  </code></pre>
  
  <p>I then run these Graph operations as below:</p>
  
  <pre><code>_, acc, loss_value = sess.run([train_op,eval_correct, loss], feed_dict=feed_dict)
  </code></pre>
  
  <p>When I run the training cycle thus, there is no optimization of the loss value, most definitely because of how I've introduced the <code>tf.stop_gradient</code> Op. </p>
  
  <p>For more details, my loss function below:</p>
  
  <pre><code>def loss(logits, labels, num_classes):
  
      logits = tf.reshape(logits, [-1, num_classes])
      #epsilon = tf.constant(value=1e-4)
      #logits = logits + epsilon
  
      labels = tf.to_int64(tf.reshape(labels, [-1]))
      print ('shape of logits: %s' % str(logits.get_shape()))
      print ('shape of labels: %s' % str(labels.get_shape()))
  
      cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, labels, name='Cross_Entropy')
      cross_entropy_mean = tf.reduce_mean(cross_entropy, name='xentropy_mean')
      tf.add_to_collection('losses', cross_entropy_mean)
  
      loss = tf.add_n(tf.get_collection('losses'), name='total_loss')
      return loss
  </code></pre>