From what I can tell only recently starting to uzse Scrapy is that alot more &quot;magic&quot;, shall we say, happens in the background so long procedures which could be a few hundred lines using bs4&#x2F;requests&#x2F;mechanize&#x2F;etc can be minimized into a lot less. Looking at Robobrowser, it seems like it will reduce some of the coding effort but not to the extent that Scrapy does.