title: How can I efficiently move many files to a new server?
body: <p>I'm switching hosting providers and need to transfer millions of uploaded files to a new server.  All of the files are in the same directory.  Yes.  You read that correctly.  ;)</p>
  
  <p>In the past I've done this:</p>
  
  <ol>
  <li>Zip all of the files from the source server</li>
  <li><code>scp</code> the zip to the new server</li>
  <li>Unzip</li>
  <li>Move directory to appropriate location
  
  <ul>
  <li>for whatever reason my zips from step 1 always bring the path along with them and require me to mv.</li>
  </ul></li>
  </ol>
  
  <p>The last time I did this it took about 4-5 days to complete and that was about 60% of what I have now.</p>
  
  <p>I'm hoping for a better way.  What do you suggest?</p>
  
  <p>File structure is hashed.  Something like this: <code>AAAAAAAAAA.jpg</code> - <code>ZZZZZZZZZZ.txt</code></p>
  
  <p>Here's one idea we're tossing around:</p>
  
  <p>Split the zips into tons of mini-zips based on 3 letter prefixes.  Something like:</p>
  
  <pre><code>AAAAAAAAAA.jpg - AAAZZZZZZZ.gif =&gt; AAA.zip
  </code></pre>
  
  <p>Theoretical Pros:</p>
  
  <ul>
  <li>could speed up transfer, allowing multiple zips to transfer at once</li>
  <li>could limit time lost to failed transfer.  (waiting 2 days for a transfer to ultimately fail is awful)</li>
  </ul>
  
  <p>Theoretical Cons:</p>
  
  <ul>
  <li>could slow down the initial zip considerably since the zip has to look up the files through a wildcard (<code>AAA*</code>), perhaps offset by running many zip threads at once, using all CPUs instead of only one.</li>
  <li>Complexity?</li>
  </ul>
  
  <p>We've also thought about rsync and scp but worry about the expense of transferring each file manually.  And since the remote server is empty I don't need to worry about what's already there.</p>
  
  <p>What do you think?  How would you do it?</p>
  
  <p>(Yes, I'll be moving these to Amazon S3 eventually, and I'll just ship them a disk, but in the meantime, I need them up yesterday!)</p>