The strict interpretation is that if &quot;robots.txt&quot; returns 403 Forbidden, it&#x27;s interpreted as &quot;deny all&quot;.  That&#x27;s what the Python library does.  We list those sites as &quot;Blocked&quot;.