title: Unclear behaviour in simple MPI send/receive program
body: <p>I've been having a bug in my code for some time and could not figure out yet how to solve it.</p>
  
  <p>What I'm trying to achieve is easy enough: every worker-node (i.e. node with rank!=0) gets a row (represented by 1-dimensional arry) in a square-structure that involves some computation. Once the computation is done, this row gets sent back to the master.</p>
  
  <p>For testing purposes, there is no computation involved. All that's happening is:</p>
  
  <ul>
  <li>master sends row number to worker, worker uses the row number to calculate the according values</li>
  <li>worker sends the array with the result values back</li>
  </ul>
  
  <p>Now, my issue is this:</p>
  
  <ul>
  <li>all works as expected up to a certain size for the number of elements in a row (size = 1006) and number of workers > 1</li>
  <li>if the elements in a row exceed 1006, workers fail to shutdown and the program does not terminate</li>
  <li>this only occurs if I try to send the array back to the master. If I simply send back an INT, then everything is OK (see commented out line in doMasterTasks() and doWorkerTasks())</li>
  </ul>
  
  <p>Based on the last bullet point, I assume that there must be some race-condition which only surfaces when the array to be sent back to the master reaches a certain size.</p>
  
  <p>Do you have any idea what the issue could be?</p>
  
  <p>Compile the following code with: mpicc -O2 -std=c99 -o simple </p>
  
  <p>Run the executable like so: mpirun -np 3 simple &lt;size&gt; (e.g. 1006 or 1007)</p>
  
  <p>Here's the code:</p>
  
  <pre><code>#include "mpi.h"
  #include &lt;stdio.h&gt;
  #include &lt;string.h&gt;
  #include &lt;stdlib.h&gt;
  
  #define MASTER_RANK 0
  #define TAG_RESULT 1
  #define TAG_ROW 2
  #define TAG_FINISHOFF 3
  
  int mpi_call_result, my_rank, dimension, np;
  
  // forward declarations
  void doInitWork(int argc, char **argv);
  void doMasterTasks(int argc, char **argv);
  void doWorkerTasks(void);
  void finalize();
  void quit(const char *msg, int mpi_call_result);
  
  void shutdownWorkers() {
      printf("All work has been done, shutting down clients now.\n");
      for (int i = 0; i &lt; np; i++) {
          MPI_Send(0, 0, MPI_INT, i, TAG_FINISHOFF, MPI_COMM_WORLD);
      }
  }
  
  void doMasterTasks(int argc, char **argv) {
      printf("Starting to distribute work...\n");
      int size = dimension;
      int * dataBuffer = (int *) malloc(sizeof(int) * size);
  
      int currentRow = 0;
      int receivedRow = -1;
      int rowsLeft = dimension;
      MPI_Status status;
  
      for (int i = 1; i &lt; np; i++) {
          MPI_Send(&amp;currentRow, 1, MPI_INT, i, TAG_ROW, MPI_COMM_WORLD);
          rowsLeft--;
          currentRow++;
  
      }
  
      for (;;) {
  //        MPI_Recv(dataBuffer, size, MPI_INT, MPI_ANY_SOURCE, TAG_RESULT, MPI_COMM_WORLD, &amp;status);
          MPI_Recv(&amp;receivedRow, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &amp;status);
  
          if (rowsLeft == 0)
              break;
  
          if (currentRow &gt; 1004)
              printf("Sending row %d to worker %d\n", currentRow, status.MPI_SOURCE);
          MPI_Send(&amp;currentRow, 1, MPI_INT, status.MPI_SOURCE, TAG_ROW, MPI_COMM_WORLD);
          rowsLeft--;
          currentRow++;
      }
      shutdownWorkers();
      free(dataBuffer);
  }
  
  void doWorkerTasks() {
      printf("Worker %d started\n", my_rank);
  
      // send the processed row back as the first element in the colours array.
      int size = dimension;
      int * data = (int *) malloc(sizeof(int) * size);
      memset(data, 0, sizeof(size));
  
      int processingRow = -1;
      MPI_Status status;
  
      for (;;) {
  
          MPI_Recv(&amp;processingRow, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &amp;status);
          if (status.MPI_TAG == TAG_FINISHOFF) {
              printf("Finish-OFF tag received!\n");
              break;
          } else {
  //            MPI_Send(data, size, MPI_INT, 0, TAG_RESULT, MPI_COMM_WORLD);
              MPI_Send(&amp;processingRow, 1, MPI_INT, 0, TAG_RESULT, MPI_COMM_WORLD);
          }
      }
  
      printf("Slave %d finished work\n", my_rank);
      free(data);
  }
  
  int main(int argc, char **argv) {
  
  
      if (argc == 2) {
          sscanf(argv[1], "%d", &amp;dimension);
      } else {
          dimension = 1000;
      }
  
      doInitWork(argc, argv);
  
      if (my_rank == MASTER_RANK) {
          doMasterTasks(argc, argv);
      } else {
          doWorkerTasks();
      }
      finalize();
  }
  
  void quit(const char *msg, int mpi_call_result) {
      printf("\n%s\n", msg);
      MPI_Abort(MPI_COMM_WORLD, mpi_call_result);
      exit(mpi_call_result);
  }
  
  void finalize() {
      mpi_call_result = MPI_Finalize();
      if (mpi_call_result != 0) {
          quit("Finalizing the MPI system failed, aborting now...", mpi_call_result);
      }
  }
  
  void doInitWork(int argc, char **argv) {
      mpi_call_result = MPI_Init(&amp;argc, &amp;argv);
      if (mpi_call_result != 0) {
          quit("Error while initializing the system. Aborting now...\n", mpi_call_result);
      }
      MPI_Comm_size(MPI_COMM_WORLD, &amp;np);
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;my_rank);
  }
  </code></pre>
  
  <p>Any help is greatly appreciated!</p>
  
  <p>Best,
  Chris</p>