title: PHP file_exists or is_file does not answer correctly for 10-20s on NFS files (EC2)
body: <p>We have an nginx/php-fpm setup on EC2 that receives file chunks to an NFS-mounted "chunk" folder (SoftNAS specifically) that is shared among multiple app servers.  We have a problem where the app checks for the existence of the file before uploading the finished file to S3, but the file check is failing even though the file is there.</p>
  
  <p>The app has a clearstatcache() in place prior to the is_file() or file_exists() (we've tried both) but the file does not become visible to the app for 10-20s.</p>
  
  <p>This is the output of some runs of that test:</p>
  
  <pre><code> app1 write timestamp 1484702190.5575
   app2 read timestamp  1484702216.0643
   25.5068 seconds
  
   app1 write timestamp 1484702229.0130
   app2 read timestamp  1484702246.0652
   17.0522 seconds
  
   app1 write timestamp 1484702265.6277
   app2 read timestamp  1484702276.0646
   10.4369 seconds
  
   app1 write timestamp 1484702286.0136
   app2 read timestamp  1484702306.0645
   20.0509 seconds
  
   app1 write timestamp 1484702314.4844
   app2 read timestamp  1484702336.0648
   21.5804 seconds
  
   app1 write timestamp 1484702344.3694
   app2 read timestamp  1484702366.0644
   21.6950 seconds
  
   app1 write timestamp 1484702374.0460
   app2 read timestamp  1484702396.0645
   22.0185 seconds
  
   app1 write timestamp 1484702404.0346
   app2 read timestamp  1484702426.0647
   22.0301 seconds
  
   app1 write timestamp 1484702434.2560
   app2 read timestamp  1484702456.1092
   21.8532 seconds
  
   app1 write timestamp 1484702466.0083
   app2 read timestamp  1484702486.1085
   20.1002 seconds
  
   app1 write timestamp 1484702496.5466
   app2 read timestamp  1484702516.1088
   19.5622 seconds
  
   app1 write timestamp 1484702525.2703
   app2 read timestamp  1484702546.1089
   20.8386 seconds
  
   app1 write timestamp 1484702558.3312
   app2 read timestamp  1484702576.1092
   17.7780 seconds
  </code></pre>
  
  <p>We've tried a number of variations on checking the file:</p>
  
  <ul>
  <li>Using the is_file and file_exists functions when checking if the file
  exists on app2. </li>
  <li>All variations of using the clearstatcache function
  before checking if the file exists on app2. </li>
  <li>Touch the file before writing to it on app1. </li>
  <li>Touch the file before checking if it exists on app2.</li>
  <li>Using different methods of writing the file from app1 and
  explicitly closing the write stream and releasing the lock on the
  file. </li>
  <li>Varying delays in between the read loops (such as no delay or
  up to a 1 second delay). </li>
  <li>Using exec to "ls" the directory after writing to it from app1. </li>
  <li>Using exec to "ls" the directory before every file exists check on app2.</li>
  </ul>
  
  <p>None of those things seems to make any difference. We didn't run extensive tests on each option, but each one seemed to take excessively long before the file exists check passed.</p>
  
  <p>There's one thing that did work. Running a loop of "ls" on app2 in a shell, the file is instantly readable by the app2 script.</p>
  
  <pre><code> app1 write timestamp 1484703581.3749
   app2 read timestamp  1484703581.3841
   0.0092 seconds
  
   app1 write timestamp 1484703638.81 00
   app2 read timestamp  1484703638.8139
   0.0039 seconds
  
   app1 write timestamp 1484703680.8548
   app2 read timestamp  1484703680.8576
   0.0028 seconds
  </code></pre>
  
  <p>So, something in the shell is correctly clearing the NFS cache, but the clear cache command in PHP doesn't appear to make any difference.</p>
  
  <p>(Edit) the code in question:</p>
  
  <pre><code>public static function get($filepath) {
  
  clearstatcache(TRUE, $filepath);
  
  if (file_exists($filepath)) {
      $instance = new static::$_class;
      $instance-&gt;init($filepath);
      return $instance;
  } else {
  
      // Sometimes a new file is not found with the first is_file() attempt.
      // Clear the stat cache and try to find the file again.
  
      clearstatcache(TRUE, $filepath);
  
      if (file_exists($filepath)) {
          $instance = new static::$_class;
          $instance-&gt;init($filepath);
          return $instance;
      }
  }
  
  Log::error("AJRFSFILE " . $_SERVER['PATH_INFO'] . " "  . $_SERVER['HTTP_DEVICE'] . " " . $filepath . " " . json_encode(stat($filepath)));
  
  return false;
  }
  </code></pre>
  
  <p>(Edit2) Turns out running exec() with an "ls" in the code successfully clears whatever file level caching is taking place at the system level, but for obvious reasons an exec() every time we do a file_exists is a sub-optimal solution.</p>