title: mirror http website, excluding certain files
body: <p>I'd like to mirror a simple password-protected web-portal to some data that i'd like to keep mirrored &amp; up-to-date. Essentially this website is just a directory listing with data organised into folders &amp; I don't really care about keeping html files &amp; other formatting elements.
  However there are some huge file types that are too large to download, so I want to ignore these.</p>
  
  <p>Using the <code>wget -m -R/--reject</code> flag nearly does what I want, except that all files get downloaded, then if  they match the -R flag, then they get deleted.</p>
  
  <p>Here's how i'm using <code>wget</code>:</p>
  
  <pre><code>wget --http-user userName --http-password password -R index.html,*tiff,*bam,*bai -m http://web.server.org/
  </code></pre>
  
  <p>Which produces output like this, confirming that an excluded file (index.html) (a) gets downloaded, and (b) then gets deleted:</p>
  
  <blockquote>
    <p>... <br>
    --2012-05-23 09:38:38--  <a href="http://web.server.org/folder/" rel="noreferrer">http://web.server.org/folder/</a> <br>
    Reusing existing connection to web.server.org:80. <br>
    HTTP request sent, awaiting response... 401 Authorization Required <br>
    Reusing existing connection to web.server.org:80. <br>
    HTTP request sent, awaiting response... 200 OK <br>
    Length: 2677 (2.6K) [text/html] <br>
    Saving to: `web.server.org/folder/index.html'
    100%[======================================================================================================================>] 2,677       --.-K/s   in 0s</p>
    
    <p>Last-modified header missing -- time-stamps turned off. <br>
    2012-05-23 09:38:39 (328 MB/s) - `web.server.org/folder/index.html' saved [2677/2677] <br>
     <br>
    Removing web.server.org/folder/index.html since it should be rejected. <br>
     <br>
    ... <br>
     <br></p>
  </blockquote>
  
  <p>is there a way to force wget to reject the file before downloading it?<br>
  Is there an alternative that I should consider?</p>
  
  <p>Also, why do i get a <code>401 Authorization Required</code> error for every downloaded file, despite supplying username &amp; password. It's like <code>wget</code> tries to connect un-authenticated every time, before trying the username/password.</p>
  
  <p>thanks, Mark</p>