Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Possible Duplicate:
Robots denied by domain is still listed in search results

I want to hide my website from all search engines for that i write robots.txt in that robots.txt i write code as

User-agent: *
Disallow: /

But still my website is in serchable why?

share|improve this question

migrated from stackoverflow.com May 21 '11 at 14:10

This question came from our site for professional and enthusiast programmers.

marked as duplicate by John Conde May 21 '11 at 15:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer 1

Although the robots.txt file the standard way to signify to search engines which areas of your sites you would like indexed and which you wouldn't, it doesn't have to be followed.

In the case of the big players, Google, Yahoo, Bing, etc, they do follow the rules.. but some of the smaller players may not and so you may appear in their indexes.

Also, if your robots.txt change is recent a search engine may take time to purge your site from their index.

share|improve this answer
{"cc_re__cc_re__cc_by_sa_3_0": ["                  <li><a href=\"//area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"//careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2015 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2015.4.24.2525\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Webmasters Stack Exchange works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>"], "copyright__copyright__copyright": ["ion.\n\n1 Answer 1\n\nAlthough the robots.txt file the standard way to signify to search engines which areas of your sites you would like indexed and which you wouldn't, it doesn't have to be followed.\n\nIn the case of the big players, Google, Yahoo, Bing, etc, they do follow the rules.. but some of the smaller players may not and so you may appear in their indexes.\n\nAlso, if your robots.txt change is recent a search engine may take time to purge your site from their index.\n\nshare|improve this answer"]}