<p dir="auto">I'd love to be able to use scrapy's <code class="notranslate">Selector</code> outside of scrapy, as an xpath (and css?) selector library for xml, html, sgml documents.<br>
I suppose I should go looking at BeautifulSoup for that, but I'm familiar with <code class="notranslate">Selector</code> and like the way it works. I don't like plain lxml's way of doing things and usually don't need any write-abilities on the tree anyway.</p>
<p dir="auto">Would it be feasible and sensible to make the selector parts of scrapy not depend on <code class="notranslate">SomeResponse</code> and expect response.url objects (but <code class="notranslate">SomeDocument</code> instead) and put it together with things such as <code class="notranslate">LxmlDocument</code>, then have some glue-code to bind it to object_ref and response objects and such in scrapy?</p>
<p dir="auto">At least in my mind this makes sense with the "librarization" goal of scrapy in mind, but maybe I'm wrong or that'd be too much of a design change when there is BeautifulSoup.</p>
<p dir="auto">p.s. before anyone comments that it is possible: I know I <em>can</em> use it outside scrapy, with something like this (but thats still having all of scrapy inside another project or rewriting a lot to factor out the selector stuff):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="namespaces = [
    ('x',    'http://www.w3.org/1999/xhtml'),
]
with open('file') as f:
    data = f.read()
xs = Selector(text=data, type='xml')
for namespace, scheme in namespaces:
    xs.register_namespace(namespace, scheme)
print(xs.xpath('/x:html').extract())"><pre class="notranslate"><span class="pl-s1">namespaces</span> <span class="pl-c1">=</span> [
    (<span class="pl-s">'x'</span>,    <span class="pl-s">'http://www.w3.org/1999/xhtml'</span>),
]
<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">'file'</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
    <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">f</span>.<span class="pl-en">read</span>()
<span class="pl-s1">xs</span> <span class="pl-c1">=</span> <span class="pl-v">Selector</span>(<span class="pl-s1">text</span><span class="pl-c1">=</span><span class="pl-s1">data</span>, <span class="pl-s1">type</span><span class="pl-c1">=</span><span class="pl-s">'xml'</span>)
<span class="pl-k">for</span> <span class="pl-s1">namespace</span>, <span class="pl-s1">scheme</span> <span class="pl-c1">in</span> <span class="pl-s1">namespaces</span>:
    <span class="pl-s1">xs</span>.<span class="pl-en">register_namespace</span>(<span class="pl-s1">namespace</span>, <span class="pl-s1">scheme</span>)
<span class="pl-en">print</span>(<span class="pl-s1">xs</span>.<span class="pl-en">xpath</span>(<span class="pl-s">'/x:html'</span>).<span class="pl-en">extract</span>())</pre></div>
<p dir="auto">I'm working on a scrapy project where a "rabbit client" and "crawl worker" work together to consume scrape requests from a queue. These requests have more configuration than a start_url - it could be something like url and a set of xpaths, or a domain-specific configuration, like site-specific product ID (from which we programmatically build the url) and optional identifiers like color, style, and size to further specify the item one wants to scrape.</p>
<p dir="auto">I ended up rolling my own generic abstract base classes to support this. These classes expose an interface that provide for crawl configuration validation and deserialization.</p>
<p dir="auto">I'm wondering if it would be desirable to have built-in support for more specific "crawl configurations" like this within the framework? If that's the case, I'd be more than happy to have a design discussion and hash out the details.</p>
<p dir="auto">I'm looking for specifically for replies from project maintainers/contributors to see whether or not this is feasible - I don't want to waste anyone's time, including mine. If it is, I'd love to tackle it myself with some design guidance, to make sure I'm working with the framework instead of against it.</p>