title: Amazon S3: How to safely upload multiple files?
body: <p>I have two client programs which are using S3 to communicate some information. That information is a list of files.</p>
  
  <p>Let's call the clients the "uploader" and "downloader":</p>
  
  <p>The uploader does something like this:</p>
  
  <ul>
  <li>upload file A</li>
  <li>upload file B</li>
  <li>upload file C</li>
  <li>upload a SUCCESS marker file</li>
  </ul>
  
  <p>The downloader does something lie this:</p>
  
  <ul>
  <li>check for SUCCESS marker
  
  <ul>
  <li>if found, download A, B, C.</li>
  <li>else, get data from somewhere else</li>
  </ul></li>
  </ul>
  
  <p>and both of these programs are being run periodically. The uploader will populate a new directory when it is done, and the downloader will try to get the latest versions of A,B,C available.</p>
  
  <p>Hopefully the intent is clear — I don't want the downloader to see a partial view, but rather get all of A,B,C or skip that directory.</p>
  
  <p><strong>However</strong>, I don't think that works, as written. Thanks to eventual consistency, the uploader's PUTs could be reordered into:</p>
  
  <ul>
  <li>upload file B</li>
  <li>upload a SUCCESS marker file</li>
  <li>upload file A</li>
  <li>...</li>
  </ul>
  
  <p>And at this moment, the downloader might run, see the SUCCESS marker, and assume the directory is populated (which it is not).</p>
  
  <p>So what's the right approach, here?</p>
  
  <p>One idea is for the uploader to first upload A,B,C, then repeatedly check that the files are stored, and only after it sees all of them, then finally write the SUCCESS marker.</p>
  
  <p>Would that work?</p>