Answers.Body: <p>I think it's there to protect those who can't have craft/ below web root. In your case it can't be found by search engines, so you <em>can</em> delete it.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-08-12T16:55:01.117Z
Answers.LastActivityDate: 2015-08-12T16:55:01.117Z
Answers.UserDisplayName: carlcs
Answers.Body: <p>The specific directive to disallow the <code>/craft/</code> directory isn't necessary if your craft folder is below the webroot.</p>
  
  <p>However, it's good practice to have a <code>robots.txt</code> for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-08-12T16:59:58.013Z
Answers.LastActivityDate: 2015-08-12T16:59:58.013Z
Answers.UserDisplayName: Michael Rog
Body: <p>I dont understand why we need the <code>robots.txt</code>file in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)</p>
  
  <p>Reason : When I initiate my domain name on the hosting it creates automatically a <strong>public folder</strong> inside the <strong>virtual host main folder</strong>.  </p>
  
  <p>So for example my hosting server make the domain name point to <code>/lamp0/web/vhosts/public/</code>. So I just have to upload <code>craft</code> and <code>public</code> folders below in <code>/lamp0/web/vhosts/</code>.<br>
  Then I just have to rename <code>public</code> in <code>htdocs</code> and no need of <code>robots.txt</code> file because <code>craft</code> folder is below domain name root.</p>
  
  <p><a href="https://i.sstatic.net/KEwFe.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KEwFe.jpg" alt="enter image description here"></a></p>
  
  <pre><code># Don't allow web crawlers to index Craft
  User-agent: *
  Disallow: /craft/
  </code></pre>
  
  <p>=========================<br>
  <strong>UPDATE</strong><br>
  OK I see now in the Craft install documentation (see below). I missed it sorry.<br>
  Perhaps would it be good to insert a <code>README.txt</code> file in the package next to robots.txt file to explain that again :)
  <a href="https://i.sstatic.net/JKDJ6.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JKDJ6.jpg" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-08-12T17:00:05.953Z
Comments.Text: Most probably don't. However, not all hosting providers allow users to edit files above the webroot (or some users just prefer both `/craft` and public files in the same directory), in which case providing a default robots.txt file preconfigured to disallow `/craft/` folder indexing is probably a good security measure (especially for those that don't know such a thing even exists). But feel free to add it as a suggestion to [http://feedback.buildwithcraft.com/](http://feedback.buildwithcraft.com/)
Comments.UserDisplayName: Douglas McDonald
Tags: <config><installation><seo>
Title: File robots.txt needed in Craft package?
ThreadText: # File robots.txt needed in Craft package?
  
  (Asked by: username_0 on 2015-08-12)
  
  I dont understand why we need the `robots.txt`file in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)
  
  Reason : When I initiate my domain name on the hosting it creates automatically a **public folder** inside the **virtual host main folder**.
  
  So for example my hosting server make the domain name point to `/lamp0/web/vhosts/public/`. So I just have to upload `craft` and `public` folders below in `/lamp0/web/vhosts/`.  
  Then I just have to rename `public` in `htdocs` and no need of `robots.txt` file because `craft` folder is below domain name root.
  
  [![enter image description here](https://i.sstatic.net/KEwFe.jpg)](https://i.sstatic.net/KEwFe.jpg)
  
  ```
  # Don't allow web crawlers to index Craft
  User-agent: *
  Disallow: /craft/
  
  ```
  
  =========================  
  **UPDATE**  
  OK I see now in the Craft install documentation (see below). I missed it sorry.  
  Perhaps would it be good to insert a `README.txt` file in the package next to robots.txt file to explain that again :)
  [![enter image description here](https://i.sstatic.net/JKDJ6.jpg)](https://i.sstatic.net/JKDJ6.jpg)
  
  **Tags:** config, installation, seo
  
  > **Comments:**
  > **username_1**: Most probably don't. However, not all hosting providers allow users to edit files above the webroot (or some users just prefer both `/craft` and public files in the same directory), in which case providing a default robots.txt file preconfigured to disallow `/craft/` folder indexing is probably a good security measure (especially for those that don't know such a thing even exists). But feel free to add it as a suggestion to [http://feedback.buildwithcraft.com/](http://feedback.buildwithcraft.com/)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2015-08-12. Score: 3 | Accepted answer
  
  I think it's there to protect those who can't have craft/ below web root. In your case it can't be found by search engines, so you *can* delete it.
  
  
  ---
  
  ### Answer by username_3 on 2015-08-12. Score: 4
  
  The specific directive to disallow the `/craft/` directory isn't necessary if your craft folder is below the webroot.
  
  However, it's good practice to have a `robots.txt` for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)
  
  
  ---
  
  