Answers.Body: <p>If you're used to another CMS such as WordPress, you're probably used to using a <code>Discourage search engines from indexing this site</code> option. As <a href="https://craftcms.stackexchange.com/a/23294/1016">Robin's answer</a> mentions, the training wheels are off. </p>
  
  <p>However, that doesn't mean you can't replicate WP's functionality.</p>
  
  <p>When you tick that box, there's a few things that happen behind the scenes:</p>
  
  <ul>
  <li>If <code>robots.txt</code> file is requested, WP auto generates a version that tells google to skip indexing the site.</li>
  <li>A <code>&lt;meta name="robots" content="noindex,follow"&gt;</code> is added to each page.</li>
  <li>XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).</li>
  </ul>
  
  <p>Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.</p>
  
  <h1>Global Lightswitch / templates</h1>
  
  <p>Create a <a href="https://craftcms.com/docs/globals" rel="noreferrer">Global</a> called <code>siteSettings</code> and add a Lightswitch with a field handle of <code>disableIndexing</code> so you can reference throughout the site.</p>
  
  <p>In the <code>&lt;head&gt;</code> section of your templates, it can be as simple as:</p>
  
  <pre><code>{% if siteSettings.disableIndexing %}
      &lt;meta name="robots" content="noindex,follow"&gt;
  {% endif %}
  </code></pre>
  
  <h1>robots.txt</h1>
  
  <p>The second piece is some Craft magic. If you don't already have a <code>robots.txt</code> file, create one inside Craft's <code>templates</code> folder or remove it/move it if it exists in your public root folder.  </p>
  
  <p>Craft is smart enough to parse your <code>robots.txt</code> file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.</p>
  
  <p>When <code>robots.txt</code> is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches <code>robots.txt</code> inside templates folder, it'll use it.</p>
  
  <p>Put this in your <code>robots.txt</code> file:</p>
  
  <pre><code>User-agent: *
  Disallow: {% if siteSettings.disableIndexing %}/{% endif %}
  </code></pre>
  
  <p>We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more <a href="https://moz.com/learn/seo/robotstxt" rel="noreferrer">options</a>.</p>
  
  <h1>Wrap up</h1>
  
  <p>That's really it.  The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit. </p>
  
  <p>Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.</p>
  
  <p>If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well.  Also consider the <a href="https://github.com/nystudio107/seomatic" rel="noreferrer">SEOMatic</a> plugin which can do all of this and more.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-26T17:16:20.997Z
Answers.LastActivityDate: 2017-12-26T17:22:24.300Z
Answers.LastEditDate: 2017-12-26T17:22:24.300Z
Answers.UserDisplayName: RitterKnight
Answers.Body: <p>Craft 3.5.10 added a new <a href="https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots" rel="nofollow noreferrer"><code>disallowRobots</code></a> config setting. If set to <code>true</code>, Craft will add a <code>X-Robots-Tag: none</code> HTTP header to frontend requests, indicating to search engines that the page should not be indexed or crawled.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-17T22:52:54.017Z
Answers.Text: Beware, on a recent build, I found this tag/header to be enabled by default, and it took some time to track down why Google was not indexing the site.
Answers.UserDisplayName: 4midori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T07:18:32.140Z
Answers.Text: The default value for the `disallowRobots` setting is (and has always been) `false`. [It *is* enabled (i.e. set to `true`) in the official Composer starter project, though](https://github.com/craftcms/craft/blob/main/config/general.php#L36) – unless the current environment does not have an environment variable `ENVIRONMENT` set to `'production'`. Probably that is what you ran into?
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-19T18:28:57.477Z
Answers.Text: Not sure. On this build, I don't think I made any changes in general.php or .env. I used an unfamiliar web host, so this setting may have been incorrect from the start.
Answers.UserDisplayName: 4midori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-14T09:08:54.903Z
Answers.LastActivityDate: 2022-03-14T09:08:54.903Z
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.Body: <p>You could also use this code in <code>robots.txt</code> file placed inside Craft's template folder.</p>
  <pre><code>{# Disallows #}
  {% if craft.app.config.env != 'production' %}
  
  {# Disallow access to everything when NOT in production #}
  User-agent: *
  Disallow: /
  
  {% else %}
  
  {# Disallow access to cpresources/ when live #}
  User-agent: *
  Disallow: /cpresources/
  
  {% endif %}
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T09:33:10.540Z
Answers.LastActivityDate: 2022-03-10T09:33:10.540Z
Answers.UserDisplayName: Samuel
Answers.Body: <p>Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create  non editable links thus you have the freedom to define everything by yourself. </p>
  
  <p>So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-26T13:41:10.693Z
Answers.LastActivityDate: 2017-12-26T13:49:10.693Z
Answers.LastEditDate: 2017-12-26T13:49:10.693Z
Answers.UserDisplayName: Robin Schambach
Body: <p>Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.</p>
  
Tags: <search>
Title: Need no search engine tracking option in Craft websites
ThreadText: # Need no search engine tracking option in Craft websites
  
  (Asked by: username_0 on 2017-12-26)
  
  Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.
  
  **Tags:** search
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2017-12-26. Score: 6
  
  If you're used to another CMS such as WordPress, you're probably used to using a `Discourage search engines from indexing this site` option. As [username_5's answer](https://craftcms.stackexchange.com/a/23294/1016) mentions, the training wheels are off.
  
  However, that doesn't mean you can't replicate WP's functionality.
  
  When you tick that box, there's a few things that happen behind the scenes:
  
  * If `robots.txt` file is requested, WP auto generates a version that tells google to skip indexing the site.
  * A `<meta name="robots" content="noindex,follow">` is added to each page.
  * XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).
  
  Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.
  
  # Global Lightswitch / templates
  
  Create a [Global](https://craftcms.com/docs/globals) called `siteSettings` and add a Lightswitch with a field handle of `disableIndexing` so you can reference throughout the site.
  
  In the `<head>` section of your templates, it can be as simple as:
  
  ```
  {% if siteSettings.disableIndexing %}
      <meta name="robots" content="noindex,follow">
  {% endif %}
  
  ```
  
  # robots.txt
  
  The second piece is some Craft magic. If you don't already have a `robots.txt` file, create one inside Craft's `templates` folder or remove it/move it if it exists in your public root folder.
  
  Craft is smart enough to parse your `robots.txt` file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.
  
  When `robots.txt` is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches `robots.txt` inside templates folder, it'll use it.
  
  Put this in your `robots.txt` file:
  
  ```
  User-agent: *
  Disallow: {% if siteSettings.disableIndexing %}/{% endif %}
  
  ```
  
  We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more [options](https://moz.com/learn/seo/robotstxt).
  
  # Wrap up
  
  That's really it. The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit.
  
  Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.
  
  If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well. Also consider the [SEOMatic](https://github.com/nystudio107/seomatic) plugin which can do all of this and more.
  
  
  ---
  
  ### Answer by username_2 on 2022-03-14. Score: 3
  
  Craft 3.5.10 added a new [`disallowRobots`](https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots) config setting. If set to `true`, Craft will add a `X-Robots-Tag: none` HTTP header to frontend requests, indicating to search engines that the page should not be indexed or crawled.
  
  > **Comments:**
  > **username_3**: Beware, on a recent build, I found this tag/header to be enabled by default, and it took some time to track down why Google was not indexing the site.  
  > **username_2**: The default value for the `disallowRobots` setting is (and has always been) `false`. [It *is* enabled (i.e. set to `true`) in the official Composer starter project, though](https://github.com/craftcms/craft/blob/main/config/general.php#L36) – unless the current environment does not have an environment variable `ENVIRONMENT` set to `'production'`. Probably that is what you ran into?  
  > **username_3**: Not sure. On this build, I don't think I made any changes in general.php or .env. I used an unfamiliar web host, so this setting may have been incorrect from the start.  
  
  ---
  
  ### Answer by username_4 on 2022-03-10. Score: 1
  
  You could also use this code in `robots.txt` file placed inside Craft's template folder.
  
  ```
  {# Disallows #}
  {% if craft.app.config.env != 'production' %}
  
  {# Disallow access to everything when NOT in production #}
  User-agent: *
  Disallow: /
  
  {% else %}
  
  {# Disallow access to cpresources/ when live #}
  User-agent: *
  Disallow: /cpresources/
  
  {% endif %}
  
  ```
  
  
  ---
  
  ### Answer by username_5 on 2017-12-26. Score: 1
  
  Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create non editable links thus you have the freedom to define everything by yourself.
  
  So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft
  
  
  ---
  
  