Permission to get the source code using spider
robots.txt file does have limits.
Its better to inform the owner of the site if you are crawling too often and read reserved rights at the bottom of the site. It is a good idea to provide a link, to the source of your content.
I am working on creating a web spider in python. Do i have to worry about permissions from any sites for scanning there content? If so, how do i get those? Thanks in advance