<h2 dir="auto">Make scrapy.selector into a separate project.</h2>
<p dir="auto">When we're scraping web pages, the most common task you need to perform is to extract data from the HTML source. There are several libraries available to achieve this: <strong>BeautifulSoup</strong>,  <strong>lxml</strong>.<br>
And although scrapy selectors are built over the lxml library, it is a better way to extract data from HTML source. In my opinion, people will prefer <strong>selector</strong> to <strong>BeautifulSoup</strong> and  <strong>lxml</strong> , if make the scrapy.selectors into a separate project,</p>
<p dir="auto">What do you think about adding response.follow_all shortcut, which returns a list of requests? This is inspired by this note in docs:</p>
<blockquote>
<p dir="auto"><code class="notranslate">response.follow(response.css('li.next a'))</code> is not valid because response.css returns a list-like object with selectors for all results, not a single selector. A for loop like in the example above, or response.follow(response.css('li.next a')[0]) is fine.</p>
</blockquote>
<p dir="auto">So instead of</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="for href in response.css('li.next a::attr(href)'):
    yield response.follow(href, callback=self.parse)"><pre class="notranslate"><span class="pl-k">for</span> <span class="pl-s1">href</span> <span class="pl-c1">in</span> <span class="pl-s1">response</span>.<span class="pl-en">css</span>(<span class="pl-s">'li.next a::attr(href)'</span>):
    <span class="pl-k">yield</span> <span class="pl-s1">response</span>.<span class="pl-en">follow</span>(<span class="pl-s1">href</span>, <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">parse</span>)</pre></div>
<p dir="auto">users would be able to write (in Python 3)</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="yield from response.follow_all(response.css('li.next a::attr(href)'), self.parse)"><pre class="notranslate"><span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-s1">response</span>.<span class="pl-en">follow_all</span>(<span class="pl-s1">response</span>.<span class="pl-en">css</span>(<span class="pl-s">'li.next a::attr(href)'</span>), <span class="pl-s1">self</span>.<span class="pl-s1">parse</span>)</pre></div>
<p dir="auto">We can also add 'css' and 'xpath' support to it, as keyword arguments; it would shorten the code to this:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="yield from response.follow_all(css='li.next a::attr(href)', callback=self.parse)"><pre class="notranslate"><span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-s1">response</span>.<span class="pl-en">follow_all</span>(<span class="pl-s1">css</span><span class="pl-c1">=</span><span class="pl-s">'li.next a::attr(href)'</span>, <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">parse</span>)</pre></div>
<p dir="auto">(this is a follow-up to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="149624295" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/1940" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/1940/hovercard" href="https://github.com/scrapy/scrapy/issues/1940">#1940</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="205755827" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/2540" data-hovercard-type="pull_request" data-hovercard-url="/scrapy/scrapy/pull/2540/hovercard" href="https://github.com/scrapy/scrapy/pull/2540">#2540</a>)</p>