Answers.Body: <p>Craft will set a <code>X-Robots-Tag: none</code> header for front end requests, if any of the following are true:</p>
  <ol>
  <li><p>The <a href="https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots" rel="nofollow noreferrer"><code>disallowRobots</code></a> general config setting has been set to <code>true</code> in your <code>config/general.php</code> file</p>
  </li>
  <li><p>The request URL includes a valid token (as per the <a href="https://craftcms.com/docs/3.x/config/config-settings.html#tokenparam" rel="nofollow noreferrer"><code>tokenParam</code></a> config setting (default is <code>'token'</code>).</p>
  </li>
  <li><p>The request URL includes a valid preview param <code>x-craft-preview</code> or <code>x-craft-live-preview</code> (note: this only applies for Craft 4.12+ and 5.4+, which are upcoming releases at the time of writing this).</p>
  </li>
  </ol>
  <p>If neither of these are the case, you might have a plugin installed that is setting the <code>X-Robots-Tag: none</code> header. For example, I believe SEOMatic will set it if Craft's <code>devMode</code> config setting is set to <code>true</code> (which should not be the case for sites in production, for a <a href="https://craftcms.com/knowledge-base/what-dev-mode-does" rel="nofollow noreferrer">plethora of reasons</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-17T22:24:52.270Z
Answers.Text: Thanks very much for your reply. It helped very much. disallowRobots wasn't set to true and I didn't have any plugins that interfered with devMode. In general.php I had 'devMode' => false when maybe it should have been $isProd (not sure if that made a difference) but what I think was the problem was in .env (which I didn't realise) I had ENVIRONMENT=dev when it should be ENVIRONMENT=production. It is now being indexed. Your pointers set me on the right path.
Answers.UserDisplayName: cannon303
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-17T23:39:07.700Z
Answers.Text: @cannon303 Ah, happy to hear that you sorted it out! Getting those environments mixed up is a common gotcha for sure.
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-21T15:00:24.190Z
Answers.Text: I like to `->disallowRobots(App::env('CRAFT_DISALLOW_ROBOTS') ?? false)` on `config/general.php` then add `CRAFT_DISALLOW_ROBOTS=true` to .env
Answers.UserDisplayName: Leo Leoncio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-21T21:49:09.650Z
Answers.Text: @LeoLeoncio Env vars prefixed with `CRAFT_` will actually override whatever is in your `general.php` file, so in practice the above is completely redundant – i.e. if you removed `disallowRobots` from your config file altogether (the default is `false`) and had `CRAFT_DISALLOW_ROBOTS=true` in your `.env` file in applicable environments, you'd have the exact same result. See https://craftcms.com/docs/5.x/configure.html#environment-overrides
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-15T21:37:10.417Z
Answers.LastActivityDate: 2024-09-02T21:17:22.550Z
Answers.LastEditDate: 2024-09-02T21:17:22.550Z
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.Body: <p>Here's how I approach this (without seomatic)</p>
  <p>.env</p>
  <pre><code>CRAFT_ENVIRONMENT=dev
  
  CRAFT_DEV_MODE=true
  CRAFT_DISALLOW_ROBOTS=true
  </code></pre>
  <p>config/custom.php</p>
  <pre><code>&lt;?php
  
  use craft\helpers\App;
  
  $isProd = in_array(App::env('CRAFT_ENVIRONMENT'), ['prod', 'production', 'live']);
  
  return [
      'isProd' =&gt; $isProd
  ];
  </code></pre>
  <p>_layout.twig</p>
  <p><code>{% set isprod = craft.app.config.custom.isProd ?? false %}</code></p>
  <p>inside <code>&lt;head&gt;&lt;/head&gt;</code></p>
  <pre><code>{{ not isprod ? tag('meta', {
      name: 'robots',
      content: 'noindex'
  }) }}
  {{ not isprod ? tag('meta', {
      name: 'googlebot',
      content: 'noindex'
  }) }}
  </code></pre>
  <p>config/routes.php</p>
  <pre><code>&lt;?php
  
  return [
      'robots.txt' =&gt; ['template' =&gt; 'routes/robots.txt']
  ];
  </code></pre>
  <p>templates/routes/robots.txt.twig</p>
  <pre><code>{% set isprod = craft.app.config.custom.isProd ?? false %}
  {% header &quot;Content-Type: text/plain; charset=utf-8&quot; %}
  {% if isprod %}
  # live
  User-agent: *
  Disallow: /admin/
  Disallow: /cpresources/
  Disallow: /cache/
  sitemap: https://my-site.com/sitemap.xml
  {% else %}
  # lower env
  User-agent: *
  Disallow: /
  {% endif %}
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-21T15:17:49.120Z
Answers.LastActivityDate: 2024-08-21T15:17:49.120Z
Answers.UserDisplayName: Leo Leoncio
Body: <p>Hello my website is not being indexed by Google. In Google Lighthouse it says:</p>
  <pre><code>Page is blocked from indexing
  
  Blocking Directive Source X-Robots-Tag: none
  </code></pre>
  <p>However I cannot find anything that could be causing the blocking. There is nothing in the .htaccess file that would point to anything and I don't have a robots.txt file and in the head of the pages I have:</p>
  <pre><code>&lt;meta name=&quot;robots&quot; content=&quot;index,follow&quot; /&gt;
  &lt;meta name=&quot;GOOGLEBOT&quot; content=&quot;INDEX, FOLLOW&quot; /&gt;
  </code></pre>
  <p>If you want to take a look the website is the-ropewalk.co.uk and is built using Craft CMS 3x. It's not a new site at all. It's about 4 years old and it was indexed at some point so something has changed. I did a full search on the site and indeed some of the templates used for the shop had robots set to no-follow but those were only used on the shop and not form the rest of the site or home page. I hyave changed those but still no joy.</p>
  <p>Is there any settings that CraftCMS has that I might have accidentally set that might mean the site is not visible to Google?</p>
  <p>What else should I check please?</p>
  <p>Any pointers would be helpful.</p>
  <p>Thank you.</p>
  
Tags: <craft3>
Title: Website not getting indexed with Google
ThreadText: # Website not getting indexed with Google
  
  (Asked by: username_0 on 2024-08-15)
  
  Hello my website is not being indexed by Google. In Google Lighthouse it says:
  
  ```
  Page is blocked from indexing
  
  Blocking Directive Source X-Robots-Tag: none
  
  ```
  
  However I cannot find anything that could be causing the blocking. There is nothing in the .htaccess file that would point to anything and I don't have a robots.txt file and in the head of the pages I have:
  
  ```
  <meta name="robots" content="index,follow" />
  <meta name="GOOGLEBOT" content="INDEX, FOLLOW" />
  
  ```
  
  If you want to take a look the website is the-ropewalk.co.uk and is built using Craft CMS 3x. It's not a new site at all. It's about 4 years old and it was indexed at some point so something has changed. I did a full search on the site and indeed some of the templates used for the shop had robots set to no-follow but those were only used on the shop and not form the rest of the site or home page. I hyave changed those but still no joy.
  
  Is there any settings that CraftCMS has that I might have accidentally set that might mean the site is not visible to Google?
  
  What else should I check please?
  
  Any pointers would be helpful.
  
  Thank you.
  
  **Tags:** craft3
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-08-15. Score: 3 | Accepted answer
  
  Craft will set a `X-Robots-Tag: none` header for front end requests, if any of the following are true:
  
  1. The [`disallowRobots`](https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots) general config setting has been set to `true` in your `config/general.php` file
  2. The request URL includes a valid token (as per the [`tokenParam`](https://craftcms.com/docs/3.x/config/config-settings.html#tokenparam) config setting (default is `'token'`).
  3. The request URL includes a valid preview param `x-craft-preview` or `x-craft-live-preview` (note: this only applies for Craft 4.12+ and 5.4+, which are upcoming releases at the time of writing this).
  
  If neither of these are the case, you might have a plugin installed that is setting the `X-Robots-Tag: none` header. For example, I believe SEOMatic will set it if Craft's `devMode` config setting is set to `true` (which should not be the case for sites in production, for a [plethora of reasons](https://craftcms.com/knowledge-base/what-dev-mode-does)).
  
  > **Comments:**
  > **username_0**: Thanks very much for your reply. It helped very much. disallowRobots wasn't set to true and I didn't have any plugins that interfered with devMode. In general.php I had 'devMode' => false when maybe it should have been $isProd (not sure if that made a difference) but what I think was the problem was in .env (which I didn't realise) I had ENVIRONMENT=dev when it should be ENVIRONMENT=production. It is now being indexed. Your pointers set me on the right path.  
  > **username_1**: @username_0 Ah, happy to hear that you sorted it out! Getting those environments mixed up is a common gotcha for sure.  
  > **username_2**: I like to `->disallowRobots(App::env('CRAFT_DISALLOW_ROBOTS') ?? false)` on `config/general.php` then add `CRAFT_DISALLOW_ROBOTS=true` to .env  
  > **username_1**: @username_2 Env vars prefixed with `CRAFT_` will actually override whatever is in your `general.php` file, so in practice the above is completely redundant – i.e. if you removed `disallowRobots` from your config file altogether (the default is `false`) and had `CRAFT_DISALLOW_ROBOTS=true` in your `.env` file in applicable environments, you'd have the exact same result. See https://craftcms.com/docs/5.x/configure.html#environment-overrides  
  
  ---
  
  ### Answer by username_2 on 2024-08-21. Score: 1
  
  Here's how I approach this (without seomatic)
  
  .env
  
  ```
  CRAFT_ENVIRONMENT=dev
  
  CRAFT_DEV_MODE=true
  CRAFT_DISALLOW_ROBOTS=true
  
  ```
  
  config/custom.php
  
  ```
  <?php
  
  use craft\helpers\App;
  
  $isProd = in_array(App::env('CRAFT_ENVIRONMENT'), ['prod', 'production', 'live']);
  
  return [
      'isProd' => $isProd
  ];
  
  ```
  
  \_layout.twig
  
  `{% set isprod = craft.app.config.custom.isProd ?? false %}`
  
  inside `<head></head>`
  
  ```
  {{ not isprod ? tag('meta', {
      name: 'robots',
      content: 'noindex'
  }) }}
  {{ not isprod ? tag('meta', {
      name: 'googlebot',
      content: 'noindex'
  }) }}
  
  ```
  
  config/routes.php
  
  ```
  <?php
  
  return [
      'robots.txt' => ['template' => 'routes/robots.txt']
  ];
  
  ```
  
  templates/routes/robots.txt.twig
  
  ```
  {% set isprod = craft.app.config.custom.isProd ?? false %}
  {% header "Content-Type: text/plain; charset=utf-8" %}
  {% if isprod %}
  # live
  User-agent: *
  Disallow: /admin/
  Disallow: /cpresources/
  Disallow: /cache/
  sitemap: https://my-site.com/sitemap.xml
  {% else %}
  # lower env
  User-agent: *
  Disallow: /
  {% endif %}
  
  ```
  
  
  ---
  
  