title: Slurm and Munge "Invalid Credential"
body: <p>I'm installing slurm for the first time.   I've installed the 19.05.1-2 tarball and used the configurator to make a very simple two node cluster.   Control node is sdc, compute nodes (running slurmd) are sdc and sdc1. Both rebuilt with Ubuntu 18.04</p>
  
  <p>I can start the controller, and the compute node sdc and also successfully submit jobs with srun.   That's great.   However, when I start slurmd on the second node, SDC1, I get:</p>
  
  <pre><code>slurmd: error: Unable to register: Zero Bytes were transmitted or received
  </code></pre>
  
  <p>That quickly led me to my munge configuration.  Munge.log on the controller (sdc) shows "Invalid credential" every second.   I triple checked that munge.key on both hosts are identical.  I verified that ntp is running too.</p>
  
  <p>So by hand I did munge -s foobar | unmunge on SDC1 and of course that worked locally.   Then I saved the munged text from SDC1 to a file on SDC and tried unmunge.  That did give me the error "Invalid credential" again.</p>
  
  <p>Because of this I uninstalled and reinstalled munge on both systems, distributed the key and repeated that test with the same result.</p>
  
  <p>I guess I'm missing something simple.  I don't know what else to do to properly install munge.</p>