i am writing a Bot that can just check thousands of website either they are in English or not.
i am using Scrapy (python 2.7 framework) for crawling each website first page ,
can some one suggest me which is the best way to check website language ,
any help would be appreciated.
If the sites are multilanguage you can send the "Accept-Language:en-US,en;q=0.8" header and expect the response to be in english. If they are not, you can inspect the "response.headers" dictionary and see if you can find any information about the language.
If still unlucky, you can try mapping the IP to the country and then to the language in some way. As a last resource, try detecting the language (I don't know how accurate this is).
python website language detection