title: Use Bash ssh to run a Java program on multiple remote machines
body: <p>I need to run a Java program on many remote machines. I'm using ssh in a loop and calling a remote script that runs the Java program.</p>
  
  <p>As you can imagine, this is used for testing a distributed system on a cluster.</p>
  
  <p>Problem is, the script hangs right after I input the password for the first ssh session. It's probably a bash error, as the Java program runs fine on local.</p>
  
  <p>The exact structure is this, a local bash script running many remote bash scripts. Each remote script compiles and runs a Java program. This Java program starts a separate thread to do some work. When a SIGINT signal is received, the Java thread is informed so it can exit cleanly.</p>
  
  <p>I made a simplified working example.</p>
  
  <p>EDIT: the code below now works (fixed for posterity)</p>
  
  <p>Please, if you want to answer, don't change the structure of the code too much, or it won't resemble the original one and I won't be able to understand what's wrong.</p>
  
  <p>Bash script that's run by hand</p>
  
  <pre><code>#!/bin/bash
  
  function startBatch()
  {
      #the problem was using -n
      ssh -f "$1" "cd $projectDir;./startBatch.sh $2"
  }
  
  function stopBatch()
  {
      #the problem was using -n
      ssh -f "$1" "pkill -f jnode_.*"
  }
  
  projectDir=NetBeansProjects/Runner
  
  #start nodes
  nodeNumber=0
  while read node; do
      startBatch "$node" "$nodeNumber"
      nodeNumber=$(($nodeNumber + 1))
  done &lt; ./nodes.txt
  
  sleep 3
  
  #stop nodes
  while read node; do
      stopBatch "$node"
  done &lt; ./nodes.txt
  </code></pre>
  
  <p>Bash script that is run by the other script</p>
  
  <pre><code>#!/bin/bash
  
  #this is a simplified working example
  myNumber=$1
  $(exec -a jnode_"$myNumber" java -cp build/classes runner.Runner "$myNumber.txt")
  </code></pre>
  
  <p>Here's a less simplified version of the above script. Check the second part of the accepted answer if you want proper logging.</p>
  
  <pre><code>#!/bin/bash
  
  batchNumber=$1
  procNumber=0
  batchSize=3
  while [ "$procNumber" -lt "$batchSize" ]; do
      procName="$batchNumber"_"$procNumber"
      #this line was no good
      #$(exec -a jnode_"$procName" java -cp build/classes runner.Runner "$procName.txt" &amp;)
      #this line works fine
      exec -a jnode_"$procName" java -cp build/classes runner.Runner "$procName.txt" 1&gt;/dev/null 2&gt;/dev/null &amp;
      procNumber=$(($procNumber + 1))
  done
  </code></pre>
  
  <hr>
  
  <p>Java Runner (the thing that starts the thread)</p>
  
  <pre><code>import java.io.File;
  import java.io.FileNotFoundException;
  import java.io.PrintStream;
  
  public class Runner {
  
      public static void main(String[] args) throws FileNotFoundException, InterruptedException {
          //redirect all outputs to a given file
          PrintStream output = new PrintStream(new File(args[0]));
          System.setOut(output);
          System.setErr(output);
  
          //controlled object
          final MyRunnable myRunnable = new MyRunnable();
  
          //shutdown the controlled process on command
          Runtime.getRuntime().addShutdownHook(new Thread() {
              @Override
              public void run() {
                  myRunnable.stop = true;
              }
          });
  
          //run the process
          new Thread(myRunnable).start();
      }
  }
  </code></pre>
  
  <p>Java MyRunnable (the running thread)</p>
  
  <pre><code>public class MyRunnable implements Runnable {
  
      public boolean stop = false;
  
      @Override
      public void run() {
          while (!stop) {
              try {
                  System.out.println("running");
                  Thread.sleep(1000);
              } catch (InterruptedException ex) {
                  System.out.println("interrupted");
              }
          }
          System.out.println("stopping");
      }
  }
  </code></pre>
  
  <hr>
  
  <p>Do not to use System.exit() in your Java program, or the shutdown hook will not be properly called (or completely executed). Send a SIGINT message from outside.</p>
  
  <hr>
  
  <p>As it was mentioned int the comments, inputting passwords can be boring. Password-less RSA keys are an option, but we can do better. Let's add some security features.</p>
  
  <p>Create the public/private key pair</p>
  
  <pre><code>ssh-keygen -t rsa
  Enter file in which to save the key (home/your_user/.ssh/id_rsa): [input ~/.ssh/nameOfKey]
  Enter passphrase (empty for no passphrase): [input a passphrase not weaker than your ssh password]
  </code></pre>
  
  <p>Add the public key to the authorized_keys file of the remote hosts, so it can be authenticated.</p>
  
  <pre><code>#first option (use proper command)
  ssh-copy-id user@123.45.67.89
  
  #second option (append the key at the end of the file)
  cat ~/.ssh/nameOfKey.pub | ssh user@123.45.67.89 "cat &gt;&gt; ~/.ssh/authorized_keys"
  </code></pre>
  
  <p>Now, if we use ssh-agent, we can make it so that the passphrase(s) will be asked only once (when executing the first command). Notice, it will ask for the passphrases (the ones inputted when creating the keys), not for the actual ssh passwords.</p>
  
  <pre><code>#activate the agent
  eval `ssh-agent`
  
  #add the key, its passphrase will be asked
  ssh-add ~/.ssh/keyName1
  
  #add more keys, if needed
  ssh-add ~/.ssh/keyName2
  </code></pre>
  
  <p>You now have a very simple yet functional testing framework for your distributed system. Have fun.</p>