title: Save model checkpoint only when model shows improvement in TensorFlow
body: <p>Do you know if there is a way to chose which model is saved when using Estimator wrapped in an experiment? Because every 'save_checkpoints_steps', the model is saved but this model is not necessarily the best.</p>
  <pre class="lang-py prettyprint-override"><code>def model_fn(features, labels, mode, params):
      predict = model_predict_()
      loss = model_loss()
      train_op = model_train_op(loss, mode)       
      predictions = {&quot;predictions&quot;: predict}
      
      return tf.estimator.EstimatorSpec(
          mode = mode,
          predictions = predictions,
          loss = loss,
          train_op = train_op,
      )
  
  def experiment_fn(run_config, hparams):
      estimator = tf.estimator.Estimator(
          model_fn = model_fn, 
          config = run_config,
          params = hparams
      )
  
      return learn.Experiment(
        estimator = estimator,
        train_input_fn = train_input_fn,
        eval_input_fn = eval_input_fn,
        eval_metrics = None,
        train_steps = 1000,
      )
  
  ex = learn_runner.run(
          experiment_fn = experiment_fn,
          run_config = run_config,
          schedule = &quot;train_and_evaluate&quot;,
          hparams =  hparams
  )
  </code></pre>
  <p>the output is as follow:</p>
  <blockquote>
  <p>INFO:tensorflow:Saving checkpoints for 401 into .\model.ckpt.</p>
  <p>INFO:tensorflow:global_step/sec: 0.157117 INFO:tensorflow:step = 401,
  loss = 2.95048 (636.468 sec)</p>
  <p>INFO:tensorflow:Starting evaluation at
  2017-09-05-20:06:07 INFO:tensorflow:Restoring parameters from
  .\model.ckpt-401</p>
  <p>INFO:tensorflow:Evaluation [1/1] INFO:tensorflow:Finished evaluation
  at 2017-09-05-20:06:09</p>
  <p>INFO:tensorflow:Saving dict for global step
  401: global_step = 401, loss = 7.20411</p>
  <p>INFO:tensorflow:Validation (step 401):
  global_step = 401, loss = 7.20411</p>
  <p>INFO:tensorflow:training loss =
  2.95048, step = 401 (315.393 sec)</p>
  <p>INFO:tensorflow:Saving checkpoints for 451 into
  .\model.ckpt.</p>
  <p>INFO:tensorflow:Starting evaluation at 2017-09-05-20:11:32</p>
  <p>INFO:tensorflow:Restoring parameters from
  .\model.ckpt-451</p>
  <p>INFO:tensorflow:Evaluation [1/1]</p>
  </blockquote>
  <p>You see that every time it saves the last model, which is not necessarily the best.</p>