title: How to configure Luigi task retry correctly?
body: <p>I am trying to configure Luigi's retry mechanism so that failed tasks will be retried a few times. However, while the task is retried successfully, Luigi exits unsuccessfully:</p>
  
  <pre><code>===== Luigi Execution Summary =====
  
  Scheduled 3 tasks of which:
  * 2 ran successfully:
      - 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask)
      - 1 MasterTask(path=/tmp/job-id-18)
  * 1 failed:
      - 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask)
  
  This progress looks :( because there were failed tasks
  </code></pre>
  
  <p>So the question is: how do I configure Luigi (I have installed version 2.3.3 with pip install) so that when a task fails once, but is then retried with success, then Luigi will exit successfully with <code>This progress looks :)</code> instead of fail with <code>This progress looks :(</code>?</p>
  
  <p>Here is a minimal scheduler and worker config I've come up with, as well as tasks to demonstrate the behavior:</p>
  
  <pre><code>[scheduler]
  retry_count = 3
  retry-delay = 1
  
  [worker]
  keep_alive=true
  </code></pre>
  
  <p>mytasks.py:</p>
  
  <pre><code>import luigi
  
  
  class FailOnceThenSucceed(luigi.Task):
      path = luigi.Parameter()
  
      def output(self):
          return luigi.LocalTarget(self.path)
  
      def run(self):
          failmarker = luigi.LocalTarget(self.path + ".hasfailedonce")
          if failmarker.exists():
              with self.output().open('w') as target:
                  target.write('OK')
          else:
              with failmarker.open('w') as marker:
                  marker.write('Failed')
              raise RuntimeError("Failed once")
  
  
  class MasterTask(luigi.Task):
      path = luigi.Parameter()
  
      def requires(self):
          return FailOnceThenSucceed(path=self.path + '.subtask')
  
      def output(self):
          return luigi.LocalTarget(self.path)
  
      def run(self):
          with self.output().open('w') as target:
              target.write('OK')
  </code></pre>
  
  <p>Example execution:</p>
  
  <p><code>PYTHONPATH=. luigi --module mytasks MasterTask --workers=2 --path='/tmp/job-id-18'</code></p>