Is Scrapy able to crawl any type of websites?
Broad Crawls
Scrapy defaults are optimized for crawling specific sites. These sites are often handled by a single Scrapy spider, although this is not necessary or required (for example, there are generic spiders that handle any given site thrown at them).
In addition to this “focused crawl”, there is another common type of crawling which covers a large (potentially unlimited) number of domains, and is only limited by time or other arbitrary constraint, rather than stopping when the domain was crawled to completion or when there are no more requests to perform. These are called “broad crawls” and is the typical crawlers employed by search engines.
These are some common properties often found in broad crawls:

they crawl many domains (often, unbounded) instead of a specific set 
of sites
they don’t necessarily crawl domains to completion, because    it
would impractical (or impossible) to do so, and instead limit the
crawl by time or number of pages crawled
they are simpler in logic    (as opposed to very complex spiders with
many extraction rules)    because data is often post-processed in a
separate stage they crawl    many domains concurrently, which allows
them to achieve faster crawl    speeds by not being limited by any
particular site constraint (each    site is crawled slowly to respect
politeness, but many sites are    crawled in parallel)

As said above, Scrapy default settings are    optimized for focused crawls, not broad crawls. However, due to its    asynchronous architecture, Scrapy is very well suited for performing fast broad crawls.
Is Scrapy framework efficient in crawling any website ? I ask this question because I found on their tutorial that they build usually regular expressions that depends on the architecture (the structure of the links) of the website to crawl it. Does this mean Scrapy is not able to be generic and crawl any website whatever the manner on which its URL are structured ? Because in my case I have to deal with a very large number of websites: it is impossible to program regular expressions for each one of them.