title: Neural Networks : designing Cost function "not differentiable" / Tensorflow
body: <p>I am testing the Tensorflow framework with a regression problem. My input has many features and my output only one. 
  For now, it is working but I think that my Cost function is not accurate for what I want, so I have pretty low accuracy.</p>
  
  <p>My goal is to predict the output within a 10% range of the actual output, that is to say for each output  :</p>
  
  <pre><code>if (y_actual - y_predicted) &lt; 0.1 * y_actual :
      cost = 0
  else :
      cost = 1
  </code></pre>
  
  <p>So for a predicted output vector of :
  <code>[130, 35, 65, 43]</code> vs an actual output vector of <code>[125, 10, 75, 40]</code>, I should get a cost of 2. (130 and 40 are in the 10%, 35 and 65 are not)</p>
  
  <p>For now, I use a squared difference.</p>
  
  <pre><code>cost = tf.reduce_sum(tf.squared_difference(y_predicted,y_)/(2*nb_batches))
  </code></pre>
  
  <p>So actually, I get a pretty good prediction for the sum of the outputs but I only have a 60% accuracy if I consider a prediction to be accurate if it is in the 10% of the actual output. 
  My accuracy is written like that :</p>
  
  <pre><code>with tf.name_scope('accuracy'):
      with tf.name_scope('correct_prediction'):
          correct_prediction = tf.less_equal(tf.abs((y_predicted-y_)), 0.1*tf.abs((y_))) # True if y_diff &lt;= 10% * y_actual
      with tf.name_scope('accuracy'): 
          accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) # count 1 if correct prediction is True ; else False ; Do the sum on all the data tested
      tf.scalar_summary('accuracy', accuracy)
  </code></pre>
  
  <p>I tried to write other cost functions such as :</p>
  
  <pre><code>cost = tf.reduce_sum(tf.cast((tf.greater_equal(tf.abs((y_predicted-y_)), 0.1*tf.abs(y_))),tf.float32)
  </code></pre>
  
  <p>But I get the error : <code>No gradients provided for any variable</code>. I think that it is because we cannot get the derivative of <code>tf.cast</code>. </p>
  
  <p>Is there a proper way to write a loss function that would fit my needs?
  I know that the loss function I want is not "derivable". Do you think that transforming it to a derivable function would be a solution ? </p>
  
  <p>I am not sure I was very clear...</p>
  
  <p>Thanks for your help !</p>
  
  <p>Paul</p>