title: SLURM slow for array job
body: <p>I have a small cluster with nodes A, B, C and D. Each node has 80GB RAM and 32 CPUs. I am using Slurm 17.11.7. </p>
  
  <p>I performed the following benchmark tests:</p>
  
  <ul>
  <li>If I run a particular Java command directly on terminal on node A, I get an result in 2minutes. </li>
  <li>If I run the same command with an "single" array job (#SBATCH --array=1-1), I get an result again in 2minutes.</li>
  <li>If I ran the same command with same parameters with an array job on slurm only on node A, I get the output in 8mininutes, that is, it is four times slower. Here, I, of course, run 31 other Java commands with different parameters at the same time. </li>
  </ul>
  
  <p>I already tried SelectTypeParameters=CR_CPU_Memory and SelectTypeParameters=CR_Core with the same result.</p>
  
  <p>Why is my array job 4 times slower? Thanks for your help!</p>
  
  <p>The header of my array job, which I submit, looks like this:</p>
  
  <pre><code>#!/bin/bash -l
  #SBATCH --array=1-42
  #SBATCH --job-name exp
  #SBATCH --output logs/output_%A_%a.txt
  #SBATCH --error logs/error_%A_%a.txt
  #SBATCH --time=20:00
  #SBATCH --mem=2048
  #SBATCH --cpus-per-task=1
  #SBATCH -w &lt;NodeA&gt;
  </code></pre>
  
  <p>The slurm.conf file looks like:</p>
  
  <pre><code>ControlMachine=&lt;NodeA&gt;
  ControlAddr=&lt;IPNodeA&gt;
  MpiDefault=none
  ProctrackType=proctrack/cgroup
  ReturnToService=1
  SlurmctldPidFile=/var/run/slurmctld.pid
  SlurmdPidFile=/var/run/slurmd.pid
  SlurmdSpoolDir=/var/spool/slurmd
  SlurmUser=&lt;test_user_123&gt;
  StateSaveLocation=/var/spool/slurmctld
  SwitchType=switch/none
  TaskPlugin=task/affinity
  
  MaxJobCount=100000
  MaxArraySize=15000
  
  MinJobAge=300
  # SCHEDULING
  FastSchedule=1
  SchedulerType=sched/backfill
  SelectType=select/cons_res
  SelectTypeParameters=CR_CPU_Memory
  
  # LOGGING AND ACCOUNTING
  AccountingStorageType=accounting_storage/none
  ClusterName=Cluster
  JobAcctGatherType=jobacct_gather/none
  SlurmctldLogFile=/var/log/slurmctld.log
  SlurmdLogFile=/var/log/slurmd.log
  
  # COMPUTE NODES
  #NodeName=NameA-D&gt; State=UNKNOWN
  NodeName=&lt;NameA&gt; NodeAddr=&lt;IPNodeA&gt; State=UNKNOWN CPUs=32 RealMemory=70363
  NodeName=&lt;NameB&gt; NodeAddr=&lt;IPNodeB&gt; State=UNKNOWN CPUs=32 RealMemory=70363
  NodeName=&lt;NameC&gt; NodeAddr=&lt;IPNodeC&gt; State=UNKNOWN CPUs=32 RealMemory=70363
  NodeName=&lt;NameD&gt; NodeAddr=&lt;IPNodeD&gt; State=UNKNOWN CPUs=32 RealMemory=70363
  
  PartitionName=debug Nodes=&lt;NodeA-D&gt; Default=YES MaxTime=INFINITE State=UP
  </code></pre>