title: Why is robots.txt being requested along with every Express GET?
body: <p>I am using the the "Getting started" tutorial for Express from the expressjs.com website.</p>
  
  <p>I am running my application (in Windows) as:</p>
  
  <pre><code>&gt; set DEBUG=express:* &amp; npm start
  </code></pre>
  
  <p>When I make a request to the server (in this example, <code>http://localhost:3000/</code>), I see in the console:</p>
  
  <pre><code>  express:router dispatching GET / +24s
    express:router query  : / +3ms
    express:router expressInit  : / +2ms
    express:router logger  : / +3ms
    express:router jsonParser  : / +3ms
    express:router urlencodedParser  : / +4ms
    express:router cookieParser  : / +3ms
    express:router serveStatic  : / +1ms
    express:router router  : / +5ms
    express:router dispatching GET / +2ms
    express:view require "jade" +3ms
    express:view lookup "index.jade" +467ms
    express:view stat "C:\mysites\app\views\index.jade" +3ms
    express:view render "C:\mysites\app\views\index.jade" +3ms
  GET / 304 545.682 ms - -
    express:router dispatching GET /robots.txt +58ms
    express:router query  : /robots.txt +2ms
    express:router expressInit  : /robots.txt +3ms
    express:router logger  : /robots.txt +3ms
    express:router jsonParser  : /robots.txt +3ms
    express:router urlencodedParser  : /robots.txt +8ms
    express:router cookieParser  : /robots.txt +1ms
    express:router serveStatic  : /robots.txt +2ms
    express:router dispatching GET /stylesheets/style.css +19ms
    express:router query  : /stylesheets/style.css +2ms
    express:router expressInit  : /stylesheets/style.css +1ms
    express:router logger  : /stylesheets/style.css +4ms
    express:router jsonParser  : /stylesheets/style.css +7ms
    express:router urlencodedParser  : /stylesheets/style.css +4ms
    express:router cookieParser  : /stylesheets/style.css +2ms
    express:router serveStatic  : /stylesheets/style.css +2ms
    express:router router  : /robots.txt +1ms
    express:router dispatching GET /robots.txt +1ms
  GET /robots.txt 304 55.631 ms - -
  </code></pre>
  
  <p>I'm trying to understand who is triggering the GET request for <code>robots.txt</code> each time I make any page/resource request. I don't believe it's the browser. And it's not in the rendered page:</p>
  
  <pre><code>&lt;!DOCTYPE html&gt;&lt;html&gt;&lt;head&gt;&lt;title&gt;Express&lt;/title&gt;&lt;link rel="stylesheet" href="/stylesheets/style.css"&gt;&lt;/head&gt;&lt;body&gt;&lt;h1&gt;Express&lt;/h1&gt;&lt;p&gt;Welcome to Express&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;
  </code></pre>
  
  <p>Is the <code>robots.txt</code> request something node.js/express is generating internally, perhaps only in this debug mode? If so, why?</p>
  
  <p>My package.json file, in case it helps:</p>
  
  <pre><code>{
    "name": "app",
    "version": "0.0.0",
    "private": true,
    "scripts": {
      "start": "node ./bin/www"
    },
    "dependencies": {
      "cookie-parser": "~1.4.3",
      "debug": "~2.6.9",
      "express": "~4.16.0",
      "http-errors": "~1.6.2",
      "jade": "~1.11.0",
      "morgan": "~1.9.0"
    }
  }
  </code></pre>