title: Creating a counter that stays synchronized across MPI processes
body: <p>I have quite a bit of experience using the basic comm and group MPI2 methods, and do quite a bit of embarrassingly parallel simulation work using MPI.  Up until now, I have structured my code to have a dispatch node, and a bunch of worker nodes.  The dispatch node has a list of parameter files that will be run with the simulator.  It seeds each worker node with a parameter file.  The worker nodes run their simulation, then request another parameter file, which the dispatch node provides.  Once all parameter files have been run, the dispatch node shuts down each worker node, before shutting itself down.</p>
  
  <p>The parameter files are typically named "Par_N.txt" where N is the identifying integer (e.g.- N = 1-1000).   So I was thinking, if I could create a counter, and could have this counter synchronized across all of my nodes, I could eliminate the need to have a dispatch node, and make the system a bit more simple.  As simple as this sounds in theory, in practice I suspect it is a bit more difficult, as I'd need to ensure the counter is locked while being changed, etc..  And thought there might be a built-in way for MPI to handle this.  Any thoughts?  Am I over thinking this?</p>