title: (Spark Schedular) What's the difference between Fair and FIFO inside a spark job pool?
body: <p>I know that for spark we could set different pools as Fair or FIFO and the behavior can be different. However, inside the fairscheduler.xml we could also set individual pool to be Fair or FIFO and I tested several times as their behavior seems to be the same. Then I took a look at the spark source code and the schedulingAlgorithm is like this:</p>
  
  <pre><code>/**
   * An interface for sort algorithm
   * FIFO: FIFO algorithm between TaskSetManagers
   * FS: FS algorithm between Pools, and FIFO or FS within Pools
   */
  private[spark] trait SchedulingAlgorithm {
    def comparator(s1: Schedulable, s2: Schedulable): Boolean
  }
  
  private[spark] class FIFOSchedulingAlgorithm extends SchedulingAlgorithm{
    override def comparator(s1: Schedulable, s2: Schedulable): Boolean = {
      val priority1 = s1.priority
      val priority2 = s2.priority
      var res = math.signum(priority1 - priority2)
      if (res == 0) {
        val stageId1 = s1.stageId
        val stageId2 = s2.stageId
        res = math.signum(stageId1 - stageId2)
      }
      res &lt; 0
    }
  }
  
  private[spark] class FairSchedulingAlgorithm extends SchedulingAlgorithm{
    override def comparator(s1: Schedulable, s2: Schedulable): Boolean = {
      val minShare1 = s1.minShare
      val minShare2 = s2.minShare
      val runningTasks1 = s1.runningTasks
      val runningTasks2 = s2.runningTasks
      val s1Needy = runningTasks1 &lt; minShare1
      val s2Needy = runningTasks2 &lt; minShare2
      val minShareRatio1 = runningTasks1.toDouble / math.max(minShare1, 1.0)
      val minShareRatio2 = runningTasks2.toDouble / math.max(minShare2, 1.0)
      val taskToWeightRatio1 = runningTasks1.toDouble / s1.weight.toDouble
      val taskToWeightRatio2 = runningTasks2.toDouble / s2.weight.toDouble
  
      var compare = 0
      if (s1Needy &amp;&amp; !s2Needy) {
        return true
      } else if (!s1Needy &amp;&amp; s2Needy) {
        return false
      } else if (s1Needy &amp;&amp; s2Needy) {
        compare = minShareRatio1.compareTo(minShareRatio2)
      } else {
        compare = taskToWeightRatio1.compareTo(taskToWeightRatio2)
      }
      if (compare &lt; 0) {
        true
      } else if (compare &gt; 0) {
        false
      } else {
        s1.name &lt; s2.name
      }
    }
  }
  </code></pre>
  
  <p>In the fairSchedulingAlgorithm, if s1 and s2 are from a same pool, the minshare, runningtask and weight should the same value and in this way we could always get the return value as false. So they are not Fair but FIFO. My fairscheduler.xml is like this:</p>
  
  
  
  <pre><code>&lt;allocations&gt;
    &lt;pool name="default"&gt;
      &lt;schedulingMode&gt;FAIR&lt;/schedulingMode&gt;
      &lt;weight&gt;3&lt;/weight&gt;
      &lt;minShare&gt;2&lt;/minShare&gt;
    &lt;/pool&gt;
    &lt;pool name="cubepublishing"&gt;
        &lt;schedulingMode&gt;FAIR&lt;/schedulingMode&gt;
        &lt;weight&gt;1&lt;/weight&gt;
        &lt;minShare&gt;0&lt;/minShare&gt;
    &lt;/pool&gt;
  &lt;/allocations&gt;
  </code></pre>
  
  <p>And spark.scheduler.mode is:</p>
  
  <pre><code># job scheduler
  spark.scheduler.mode              FAIR
  spark.scheduler.allocation.file   conf/fairscheduler.xml
  </code></pre>
  
  <p>Thanks for your help!</p>