I need to do lot html parsing / scraping /search engine /crawling.
There are many libraries currently like Scrapy, Beautiful Soup, lxml , lxml2 requests, pyquery.
Now i don't want to try each of these and then decide. basically i want to follow on one and then study in detail and then use that most often.
So which library should i go for which can perform all function mentioned above. Even though there may be diff solutions for diff problems. But i want onelibrary which could do all things even though it takes time to code but should be possible
Is it possible to do indexing in lxml? Is PyQuery same as lxml or its different?
I'm using Beautiful Soup and am very happy with it. So far it answered all my scraping needs. Two main benefits:

It's pretty good at handling non-perfect HTML. Since browsers are quite lax, many HTML documents aren't 100% well-formed
In addition to high-level access APIs, it has low-level APIs which make it extendible if some specific scraping need isn't directly provided
If i have to choose only one html scraping library for python, which should i choose