Does Scrapy 'know' when it has crawled an entire site?
Scrapy uses a link follower to traverse through a site, until the list of available links is gone.  Once a page is visited, it's removed from the list and Scrapy makes sure that link is not visited again.
Assuming all the websites pages have links on other pages, Scrapy would be able to visit every page of a website. 
I've used Scrapy to traverse thousands of websites, mainly small businesses, and have had no problems.  It's able to walk through the whole site.
I have used Beautiful Soup with great success when crawling single pages of a site, but I have a new project in which I have to check a large list of sites to see if they contain a mention or a link to my site. Therefore, I need to check the entire site of each site.
With BS I just don't know yet how to tell my scraper that it is done with a site, so I'm hitting recursion limits. Is that something Scrapy handles out of the box?